21 Feb 2025

Cloudflare for the Selfhoster

If you selfhost any applications and have wondered how you can best access those applications from outside your network, you’ve undoubtedly come across Cloudflare. Cloudflare offers two services in particular that might be attractive to homelabbers and selfhosters, reverse proxying and “tunnels”.

Both of these offer some degree of benefit; proxying potentially offers a degree of Denial of Service (DoS) 1 protection and use of their Web Application Firewall (WAF), and the tunnel additionally adds the benefit of circumventing the host’s firewall and port-forwarding rules. To sweeten the deal, both of these services are offered “for free”, with the proxying actually being the default option on their free plans.

Lets examine these benefits briefly and see why they’re as popular as they are.

Advantages


DoS Protection

By virtue of being such a massive network, Cloudflare’s proxies is able to absorb a massive number of requests and block those requests from reaching their customer. You can think of it as a dam, when a massive flood of requests come in, Cloudflare is in front of the customer holding back the waters.

WAF

Cloudflare’s WAF allows customers to set firewall rules on the proxy itself, which can block or limit outside requests from ever reaching the customer, similar to the DoS protection.

Bypassing Firewalls (tunnel)

Many firewalls are configured to allow outgoing connections and block incoming connection. This is great for security, since it prevents others from accessing your stuff from the public internet, but this can be a problem if you want people to access your website or services. Cloudflare’s tunnel service takes advantage of the “allow outgoing” rule to establish a connection from the host server to Cloudflare, and then allowing incoming requests to tunnel through this connection and access the host server.

Bypassing Port-Fowarding rules (tunnel)

If you’re hosting a service from behind an IPv4 address, there’s a good chance that you’re using a technology called Network Address Translation (NAT) 2. This allows you to have multiple systems running behind a single public IP address. This concept can be taken further by Carrier Grade (CG) NAT, where the ISP applies NAT rules to their customers. The problem with NAT is that it becomes difficult to make an incoming request to a machine behind NAT. To an outside system, all of the machines behind the NAT layer appears to have a single address. To partially overcome this, a rule needs to be configured that says “when a request comes in for port X, forward it to private address Y, port Z.” This allows, in a limited way, an outside connection to make use of the NAT address and a particular port to send traffic to a specific machine behind the NAT layer. In the case of CGNAT however, these rules would need to be created by the ISP, a service which most do not offer. Similar to taking advantage of the outgoing firewall rules, a tunnel also uses outgoing ports to avoid the need to create specific port forwarding rules and then directly tunnels a list of ports straight to the host machine.

On the surface, this all sounds like a great product; it allows smaller customers to selfhost even if in a sub-optimal environment.



Considerations


Cloudflare’s proxy and tunnel models both have several aspects, however, which might make some want to reconsider their use.

DoS Protection

DoS attacks can vary in size and scope and their impact will depend not only on the volume of the attack, but also on the capabilities of the targeted system. For example, for a small home server, a few hundred connections per second might be enough cause a denial of service. On the extreme opposite, Cloudflare’s servers routinely handle an average of 100,000,000 requests per second without issue. With those numbers in mind, a DoS on a small home server might be so small that it goes unnoticed by Cloudflare’s protections. I do have to say “might” here, though, because I could not find a definitive answer to how Cloudflare determines what a DoS attack looks like. I would expect, however, that the scale of a home server would be so small compared to the majority of Cloudflare’s customer base that an effective DoS attack would not look significantly different than “normal” traffic.
Lastly, and quite importantly, this protection only exists for attacks targeted at a domain name. Domains exist to make the internet easier for humans, for a computer script or bot, an IP address is far simpler. If an attacker targets the hosts IP address directly, that attack will completely bypass whatever protection Cloudflare was providing. If you’re behind CGNAT, this means the attack will target the ISP, but if you have a publicly addressable IP address, that attack is directly targeting your home router.

Security

Cloudflare offers these services for free, and as the saying goes: “If you are not paying for it, you’re not the customer; you’re the product.” Cloudflare advertises that their proxy and tunnel services can optionally provide “end-to-end” (e2e) encryption. In the traditional and commonly used definition, this means that traffic is encrypted by the client (your or your users) and decrypted by the host (your server). Under the normal definition, no intermediary device can decrypt and read the traffic.

Cloudflare, however, uses the term a little differently, as you can see in this graphic.

cloudflare’s “end-to-end” encryption

Instead of providing traditional e2e, Cloudflare acts as a “Man in the Middle” (MITM), receiving the traffic, decrypting it, analyzing it, and then re-encrypting it before sending it along. Cloudflare does this in order to provide their services; they collect and store the unencrypted data in order to apply WAF rules, analyze patterns, and so on.
Now, Cloudflare is a giant company with billions in contracts; contracts they could potentially lose if they were found to be misusing customer data. They wouldn’t benefit by leaking your nude photos from your NextCloud instance or exposing your password for Home Assistant, but you should understand that by giving them the keys to your data, you are placing your trust in them. This MITM position also means that, theoretically, Cloudflare could alter your content without the you (or your users) knowing about it. Normally this would cause a modern browser to display a very large “SOMEONE MIGHT BE TRYING TO STEAL YOUR DATA” warning, but because you are specifically allowing Cloudflare to intercept your data, the browser has no way of confirming whether the content actually came from your server or Cloudflare themselves.
Cloudflare does have a privacy policy, which explains exactly how Cloudflare intends to use your data, and a transparency report, which is intended to show exactly how many times each year Cloudflare has provided customer data to Government entities.

The warning you would normally get during a MITM attack:
SSL Warning

Content Limitations

Lastly, Cloudflare, by default, only proxies or tunnels certain ports. If you want to forward an unusual port through Cloudflare, like 70 or 125, you would need to use a paid account or install additional software. Cloudflare's Terms of Service also limit the type of data you can serve through their free proxies and tunnels, such as prohibiting the streaming of video content on their free plans.

Alternatives


Are there other ways to get similar benefits?

  • If only a limited number of users need access to your server or network and you have a public IP address, the best solution by far is to use a VPN service, such as Wireguard. Wireguard allows you to securely access your network without exposing any attack surface for bots or malicious actors.
  • If you don’t have a public IP, there’s a service called “Tailscale” which uses the same outgoing-connection trick to bypass firewalls and CGNAT, but instead of acting as a MITM privy to all of your traffic, Tailscale simply coordinates the initial connection and then allows the the client and host to establish their own secure, encrypted connection.
  • If you do want to expose a service to the world (like a public website) and you have a public IP address, you can simply forward the needed port/s from your router to the server (typically 443 and possibly 80).
  • If you want your service to be exposed to the world , but are stuck behind CGNAT, then a Virtual Private Server (VPS) is a potential option. These typically have a small cost, but they provide you with a remote server that can act as a public gateway to your main server. They can also provide a degree of DoS protection, since they’re using a much larger network and you can simply turn off the connection between the VPS and your server until things calm down.

Closing thoughts

Cloudflare offers some great benefits, but if you're particularly security-minded, you may want to look into alternatives. Even though Cloudflare is trusted by numerous customers around the world, you still have to decide if you want to trust them with your data. While there are other alternatives, though, Cloudflare's offerings are mostly free and comparatively easy to use.




1. DoS or a "Denial-of-Service (DoS) Attack" is a cyber act in which an attacker floods a server with internet traffic to prevent legitamte users from accessing the services. Cloudflare offers an example of one type of DoS attack here.

  1. NAT translates private IP addresses in an internal network to a public IP address before packets are sent to an external network.

19 Feb 2025

Inspect What You Expect

That's a phrase I heard for the first time when I became a manager; my boss would tell me I needed to "inspect what I expect." To be honest, I don't think I fully understood the phrase until a lot later. In the context of leadership, it means that if you have certain expectations for your team, you should actively verify that those expectations are being met and investigate the cause if they're not. Don't wait until eval season to say "You failed to meet expectations", instead, review expectations regularly, and if someone isn't meeting them, it's your job to figure out why. Have you defined the expectations in a way that they understand? Are your expectations realistic? Did you provide the resources they needed to meet the exceptions? Did you provide guidance when needed? And so on.

You might be asking: "Sure, but why is this on a blog about computers and stuff?"

Expectations


You might have the expectation that your network is secure. You don't remember adding any firewall rules or forwarding any ports, or maybe you followed some YouTube tutorial and they told you to proxy your incoming connections through Cloudflare. Maybe you set up a "deny all" line in that NGINX config or you only access your network through a VPN. You expect that your network is safe from outside actors.

To use a real world example, my father set up several Lorex brand CCTVs on his home network to save video to an NVR. He configured a VPN to allow secure remote access to the NVR and assumed that everything was safe. He didn't create any firewall rules or port forwarding to allow access by anything other than the VPN. He expected that this was safe, and in theory, it should have been.

Inspection


We traveled together for a holiday and while drinking coffee one morning, I showed him Censys. Censys is search engine, but unlike Google or Bing, which search for webpages, Censys searches for the actual devices. It indexes the IP addresses of everything on the internet and everything about them. You can write queries to search this index for almost anything you can think of.

For a quick demonstration, I just searched for his home IP address, and we were both surprised when we saw the results:
not actually his results

Umm... What?

For those who might not know RTSP stands for Real-Time Streaming Protocol. It is a protocol designed to stream video over a network, commonly used for CCTV cameras. The HTTP port was the HTML login screen for one of his cameras. We clicked it and I sarcastically tried a default login (admin, admin) and we found ourselves staring at his basement.
As far as he knew, he took all the appropriate steps to secure the devices on his network and expected that nothing was unsafely exposed, but he hadn't inspected that expectation. We found that the camera was factory configured to expose itself via uPnP, a technology which allows devices to request changes to port-forwarding and firewall rules without user involvement. This is supposed to allow for easy set-up by inexperienced users, but it can also create significantly compromise security without the use knowing about it. In our case, my father is not an inexperienced user, he's been a computer engineer since the 80s and has even worked for one of the major producers of networking equipment. He took all the right steps to get his expected result, he just hadn't inspected it ensure that his expectation was being met.

Inspect What You Expect


Censys can be a great starting point when evaluating your network by helping you to understand what you have exposed to the internet. Censys queries can be as simple as an IP address if you just want to see a single point, or they can search broadly for very specific things.

Here are the results of a very simple query looking for exposed FTP servers based out of japan:
japan

There's a lot of FTP servers, obviously, but did you notice that SSH server on port 10022? Some people expect that services will be hidden if they run them on non-standard ports, but don't inspect to see if that's actually the case. Here, we can see that the SSH server is still quite visible, despite being on a non-standard port, just like those non-standard HTTP servers on the other entries. Clicking into an entry will provide even more information, like the software versions, request responses, and so on.

Through Censys, I realized that I was running an older version of Nginx than I thought I was, and that this older version had a number of vulnerabilities that were patched in later versions. I expected that I was running the a current version, but my inspection showed me otherwise.

Final thoughts


While a tool like Censys isn't the only tool you should use to inspect your security expectations, it's a great starting point, since it can show you what your network looks like to the internet. t's also a fun tool to use to explore the internet from a different angle. Instead of just searching the surface of the web for youtube videos and news stories, try searching deeper for Roombas, smart lights, or security cameras.

The important takeaway, though, is that just because you think that something is working how you expect it to, doesn't always mean that it is.
The only way to know for sure is to inspect what you expect.

17 Feb 2025

Ramblings on my digital life

Are you in control of your digital life, or is it controlling you?
Who decides what you see online?
What information is being collected about you?

Like many, I'm going through a phase of re-evaluating my interaction with social media and to some extent, computer technology as a whole. I'm not entirely "degoogling," like some are trying to do, rather I'm trying to tailor my digital life to me instead of the other way around. I don't want to be fed a "curated" stream content that some algorithm has decided that I'm most likely to "engage" with, but rather content I want contant that I want to see, in a format that I've chosen to see it in.

I want control.



Blogs

One step towards achieving this goal is what you're reading now. This blog. I decided on the software. I built the server that it's running on. I picked the ugly colors (unless you're reading this via gopher). I can delete posts, create new posts, or destroy the blog entirely. There are no trackers, ads, AI, or other garbage unless I decide that there should be. This doesn't get the readership that I would if I were writing this on Facebook, but the few people who do read it are reading it because they want to, and that's what's important.

I've also started following other blogs, too, some of which are listed in the "Links" section in the menu to the left. Reading personal blogs is great, because you're just reading what a person wrote. There's no algorithm, it's just delivering whatever they typed straight into your eyeballs.

Websites

I maintain my personal homepage and I've been visiting other personal websites. While I personally self-host my homepage, sites like Neocities are making it easy for individuals to create personal webpages without needing to manage a webserver themselves. Searching for new personal homepages can be a little difficult compared to massive SEO'd sites, but services like "webrings" can help you find other similar homepages and provide plenty of rabbit holes to travel down. My homepage is part of the "No AI" webring, whose membership consists of personal pages free of AI generated content.

Email

I still use gmail, it's true, but I now interact with it through Thunderbird. I know that doesn't remove all of the Google tracking and whatnot, but it does allow me to access my email in a presentation that I prefer, and it allows me to use PGP/GPG, which does prevent Google from reading my emails, if it's actually used. Unfortunately I don't receive many PGP encrypted emails, but I've published my public key and my domain supports WKD, so it's easy to access if anyone wants to use it. My address is just "me" @ this domain, by the way, if anyone wants to send something my way.


RSS Feeds

Another advantage to Thunderbird is the built-in RSS reader. Instead of visiting, say, reddit.com, I can import my favorite subreddits directly into Thunderbird as RSS feeds, as well as following other blogs, podcasts, forums, and so-on. No ads, no trackers, just the content I actually want to see.


Fediverse

Of course the Fediverse has to play a role. The "fediverse" is a network of social media applications that can all communicate with each other. It's sort of like if Facebook, Reddit, Snapchat, Youtube, ~~Twitter~~ "X", and so on were all interconnected… Except that each service is actually composed of thousands of individually-owned servers instead of a massive server farm owned by a single entity. If I decide to take down my server, my posts will eventually disappear but the fediverse will live on. No single entity controls the fediverse. Owning the server also means that I control the server. I can decide not to federate (interact with) other servers or limit which servers I want to federate with. I'm in control of my own social media.

SearXNG

"SearXNG is a metasearch engine, aggregating the results of other search engines while not storing information about its users." Essentially, instead of directly using Google or Bing, I search via my own SearNGX server which will create a new "identity" for every search, aggregate the results from dozens of other search engines, and then strip out all of the ads and sponsored results. That means I can search for things like "political news" and get results from all across the internet without getting blasted with advertisements for MAGA baseball caps for the next month.

Last thoughts

Breaking away from major social media and massive corporate entities isn't easy (or even totally feasible) I do feel a lot better knowing that I have al little more control over this aspect of my digital life.

14 Feb 2025

Lynx - A Text Based Web (and Gopher) Browser

What is it?

The Lynx Browser (not to be confused with the later Links Browser), is the oldest actively maintained web browser, initially developed in 1992 and still being maintained today. What makes lynx look a bit different from other modern browsers is the fact that even in 2025, it can only display basic text. The lynx browser ignores all of the ads, pictures, JavaScript, fancy formatting, and annoying infinitely-scrolling slop, to just delivery the content you want to read. Not only does this reduce distractions, but it's also great to use if your internet connection has limited bandwidth.

Using lynx

Lynx is included in the repos of most Linux distributions, so installing it is just a matter of running install lynx via your favorite package manager. Lynx runs in the terminal emulator, and can be started by the command lynx or you can directly open a specific website with lynx {website you want}.

Here's an example of Google.com:

google.com

And here's Wikipedia:

Wikipedia

The formatting has obviously changed quite a bit, but all of the content is still there. This makes lynx great for sites where you just want to be able to read the content quickly, without distractions. Lynx also supports the gopher protocol, and since gopher is a text-based service to begin with, browsing the gopherspace from lynx feels completely natural.

Super Dimensional Fortress's User's Gophersites:

Super Dimensional Fortress

Pitfalls

There are, however, a few downsides to accessing the modern W3 through a text-only browser. The things that many of us would like to avoid (ads, JavaScript, endlessly scrolling slop, etc) have been so deeply entrenched in some sites that the they simply can't function without it. You might think that this could be a great way to revisit pages that you used to enjoy, like Facebook, but here's how the modern Facebook (even m.facebook) looks on Lynx:

facebook

In fact, even some some simple websites might not display well in lynx built if they're built with certain older formatting tools, like frames. Here's my own website as an example. It's worth noting, though, that while my site doesn't look correct without displaying frames, it can still be fully navigated by using the FRAME: links at the top:

k3can's homepage

Thoughts

So, while lynx is unlikely to fully replace your graphical browser on the modern web, it's still surprisingly useful for focused reading, navigating gopherspace, and for situations with limited bandwidth. Installing lynx is simple and the entire browser is only 6 MiB in size, so it's a great tool to have on your system.

4 Feb 2025

New look!

I know absolutely nothing about CSS.

My very first webpage was coded in straight HTML (as is my current homepage), and when I decided I wanted something fancier, WYSIWYG editors were becoming popular, so I sort of skipped over the whole "learning css" thing.

This blog is "powered by DotClear" which offers a simple way to import "themes". The Themes actually provide the color themes and the structure of the blog. The theme I'm using here is called "atopa", but I wanted something a little bit different. I poked and prodded around in the mysterious stylesheet code, made some educated guesses, and ended up with what you see now. It's inspired by the color scheme of the Counter Strike mod for the first Half Life game. In addition to being a mod I played a ton when it came out (20 years ago, oh my!), I also just enjoy the subdued earthy tones (my closet is full of greens, browns, tans, and greys), and I find that it's easier on the eyes than bright white, while still being more colorful than the typical "dark mode" website. I also tweaked the spacing a bit, since I felt that the default spacing squished all of the content into the middle of the screen and left huge empty gaps on each side.

Anyway, if you don't like the colors, this blog is also available via RSS feed and gopher, both of which will leave the coloring up to the client you're using.

1 Feb 2025

A rambling on the topic of generative AI.

While trying to write the script mentioned in a prior post, I decided that I would use an XML parsing module of some sort rather than writing my own.  A quick google search turned up a number options, but I wasn't quite sure which would be best for my use-case.  I knew that I wanted to specifically read RSS feeds, so using a general purpose XML parser was probably overkill if there was a more RSS-focused option available. I found several RSS-focused modules to choose from, including XML::RSSLite; which sounded promising as a light-weight RSS-specific module. 

The documentation was rather sparse, though, and I thought I might get a better idea of how to use it if I could see some examples. When a google search didn't turn up much in the way of results, I decided I would just ask an AI to create an example for me.

In its example, it makes this call:

my $rss = XML::RSSlite->new();

...Which might look reasonable on the surface, but it didn't actually work. Now, instead of understanding how the module works by reviewing a working example, I get to learn by trying to troubleshoot something else's broken code. After actually checking the documentation, it seems that "XML::RSSlite" isn't an object and the module never uses a "new" method.   After I pointed this out, the AI gave it another go:

my $rss = XML::RSSlite->parse_url($feed_url);

Closer, I guess?  It still doesn't work, though, because it's still trying to call on objects and functions that don't exist. I pointed that out again and let it try one more:

my $rss = XML::RSSlite->new();

Really?

Despite trying to prompt it into a correct answer, it kept admitting that each answer was wrong and then produced the same incorrect code again and again.

For reference, here's how the module is intended to be used:

use XML::RSSLite;
parseRSS(\%result, \$content);
Yep. That's it. Super simple. One function that takes two arguments: a reference to an input scaler and an output hash.
 
It's a little baffling that the AI had this much trouble with such a simple request, but my guess is that it comes down to one fact: XML::RSSlite doesn't get a lot of use.
An AI "learns" by slurping up other examples and building a model based on what it sees. It doesn't actually "understand" any of it, it just knows that things generally happen a particular way.  Essentially, it just constructed something that looked like working code, without any knowledge of whether it would function or why. Even when it was told that it was wrong, it still produced the wrong code because it looked the most "right". It doesn't read man pages or reference documents to better understand how to answer a problem, it reads man pages to understand how a man page should look and reads answers to understand how an answer should look.
 
And I feel like this is really where the "AI is going to take over the world" thing gets a bit silly. The AIs of today can't create new things, they can only create variants of things they've already seen. They can write a story because they know what a story should look like.
They can create an image of a cat riding a Craftsman lawnmower, because they know what cats and Craftsman lawnmowers should look like.
 
I asked a Generative AI to "create an image of something entirely new, that has no analogs in the current world."
It produced an admittedly, very pretty, picture of a cave with a river running through it:
 
 
I'm sure that exact cave doesn't exist in the known world, but it's far from an entirely new concept of a thing.
On the other hand, human artists like H R Giger can create surreal nightmare bullshit that looks like nothing you've even seen or ever want to see again. 
 
I guess my conclusion is that If you're going to fear anything in the current world, it should be us humans.

31 Jan 2025

Short updates - new keyboard and SSDs

New Keyboard

I picked up a Feker Alice98 in an attempt to get something a little more "ergonomic" than my previous keyboard. It's an "Alice" layout, with the interesting distinction of having two "B" keys.
This keyboard is also compatible with VIA, making keymapping, backlighting, and macros easy to manage. This keyboard isn't listed on VIA's website, though, so it requires manually importing the keyboard definition file (attached to this post). Hopefully VIA will add it officially, eventually.  

"New" SSDs for the cluster

I also picked up some gently-used Intel TLC SSDs for my main Proxmox nodes.

I had noticed before that the IO delay [1] on my nodes would creep in to the 80%+ range when running any reasonably write-intensive task, like a system update. I also received feedback that this blog seemed slow, which I suspected might be caused by the same issue.  While the consumer QLC SSDs I was previously using seem fast for normal desktop use, their short-comings become quite noticeable when running multiple VMs. Here's a screen shot of the IO delay during fairly normal use before, compared to the the IO delay now while running updates:

(blue is the IO delay)

While the old ones would routinely creep in to the 50% or higher range while running fairly simple tasks, the new SSDs peaked around 5%, even when under load.

 

Note(s)

  1. ^ Amount of time that the CPU hangs idle while waiting for IO tasks (reading or writing to a drive) to complete.

26 Jan 2025

Blog Mirroring to Gopher

This blog is also available via gopher! And here's how:

Gophernicus

As I mentioned in my previous post, my gopher server of choice is Gophernicus. One of the benefits of Gophernicus is that it will automatically generate gophermaps based on the files in it finds in a directory. This means that adding entries is as simple as dropping a file into the directory. The next time that server is accessed, the new file will appear automatically.

Mirroring

All that remains is finding a way to easily add files to the gopher directory. Since I already have this blog, I decided the first thing I wanted to do was to mirror these posts into the gopherspace. This blog uses Dotclear, which provides a plugin to access an RSS feed for the blog. By fetching (and then parsing) this feed, I can export the contents into text files accessible to gopher. I wrote a perl script to accomplish this and created systemd units to execute that script on a recurring schedule to pull in new entries. The full source code is available on my github. The script uses LWP:Protocol:https to fetch the RSS feed and XML::Feed to extract the title, date, and body of each entry. The date and title are used as the file name, and the body is reduced down to plain text and then written to the file.

If you'd like to use the script, it should, probably, maybe work with other RSS and ATOM feeds, but some feeds can be a bit loosey-goosey about how they handle their XML, so no guarantees.

18 Jan 2025

Let's Gopher!

I've decided it's time to get my gopher server running again, and wanted to outline the basic steps for anyone else looking to dive into the gopherhole.

install

I personally like gophernicus, a "modern full-featured (and hopefully) secure gopher daemon." Assuming that you're running a Debian server, we can install a simple gopher server via apt:

apt install gophernicus

installing screenshot

This should also install a socket unit and create a service template for gophernicus.

config

Gophernicus defaults to serving from either /var/gopher/ or /srv/gopher. On my install, the default config file is stored at /etc/defaults/gophernicus. I feel like /etc/defaults/ isn't used very often these days, but the intention is to provide a simple way for a developer to provide a default configuration for their application. We can create a new config somewhere reasonable, like /etc/gophernicus or just modify the existing file in /etc/default/ like a lunatic. I'm choosing the latter, of course.

The important thing to add is a hostname, but we can also turn off any features we don't want.

My config looks like OPTIONS=-r /srv/gopher -h gopher.k3can.us -nu.

The -nu option just disables the "personal" gopherspace (serving of ~user directories). Unless you know what this does and intend to use it, I'd suggest disabling it.

testing

We should now be able to start the service and access our empty gopherhole via systemctl start gophernicus.socket, which will create an instance of the gophernicus@.service unit. We can run a quick test by creating a file to serve and viewing the gophermap. To create the test file: touch /srv/gopher/test.txt, and then we can fetch the gophermap via telnet [ip address]-70.

 Trying 192.168.1.5...
Connected to 192.168.1.5.
Escape character is '^]'.

i[/]    TITLE   null.host   1
i       null.host   1
0hello-world.txt                       2025-Jan-18 09:47     0.1 KB /test.txt   gopher.k3can.us 70
.
Connection closed by foreign host.

That little jumble of text is our gohpermap. Lines starting with i indicate "information", while the 0 indicates a link to a text file. Gophernicus creates this map automagically by examining the content of the directory (although it also provides the option of creating a map by hand). To add files and folders, we can simply copy them into the /srv/gopher and gophernicus will update the map to include the new files.

From here, if we want to expose this publicly, can simply route/port-forward through our router.

In my case, though, I'm going to need to configure a few more components before I open it up to the public... First, I use (apparmor)[https://apparmor.net/] to limit application access, and second, my webserver lives behind a reverse proxy.

apparmor

For apparmor, I created a profile for gophernicus:

include <tunables/global>
# AppArmor policy for gophernicus
# by k3can

/usr/sbin/gophernicus {
  include <abstractions/base>
  include <abstractions/hosts_access>
  network inet stream,

  /etc/ld.so.cache r,
  /srv/gopher/ r,
  /srv/gopher/** r,
  /usr/bin/dash mrix,

}

This profile limits which resources gopernicus has access to. While gophernicus should be fairly secure as is, this will prevent it from accessing anything it shouldn't on the off-chance that it somehow becomes compromised. Apparmor is linked above if you want to get into the details, but I'm essentially telling it that gophernicus is allowed to read a number of commonly-needed files, run dash, and access its tcp stream. Gophernicus will then be denied access to anything not explicitly allowed above.

nginx

Lastly, to forward gopher through my reverse proxy, I added this to my nginx configuration:

#Gopher Stream Proxy

 stream {
     upstream gopher {
         server 192.168.1.5:70;
     }

     server {
              listen 70;
              proxy_pass    gopher;
     }
 }

Since nginx is primarily designed to proxy http trafic, we have to use the stream module to forward the raw TCP stream to the upstream (host) server. It's worth noting that as a TCP stream, ngnix isn't performing any virtual host matching, it's simply passing anything that come into port 70 on to the uptream server. This means that while I've defined a specific subdomain for the host, any subdomain will actually work as long as it comes into port 70; gopher://blog.k3can.us, gopher://www.k3can.us, and even gopher://sdkfjhskjghsrkuhfsef.k3can.us should all drop you into the same gopherhole.

access

While telnet will show you the gophermap, the intended way to traverse gopher is through a proper client application. For Linux, Lynx is a command-line based web-browser/gopher client available in most repos, and for Android, DiggieDog is available through the PlayStore.

next steps

Now all that's for me to do is add content. I used to run a script that would download my Mastodon posts and save them to a "phlog" (a gopher blog), and I could potentially mirror this blog to there, as well. That helped me keep the content fresh without needing to manually add files. I haven't quite decided if I want the gopherhole to primarily be a mirror of my other content, or if I want to be more intentional with what I put there.

Besides figuring out content, I'm also curious about parsing my gophernicus logs through Crowdsec. Unsurprisingly, there's not currently a parser available on the Crowdsec Hub, so this might take a little tinkering...

12 Jan 2025

Wireguard - Securely connecting to your network

Background

After setting up my home server, I found that I wanted a way to securely access and manage my system when I was away from home. I could have opened the SSH port to the internet, but the idea of having such a well-known port exposed to the internet made me wary not only of attempts to gain unauthorized access, but even just simple DOS attacks. I decided the far better option was to use a VPN, specifically, Wireguard.

Wireguard is a simple, lightweight VPN, with clients for Linux, Android, Windows, and a handful of other operating systems. In addition to the security offered by its encrypted VPN tunnel, Wireguard is also a "quiet" service; that is, the application will entirely ignore all attempts to communicate with it unless the correct encryption key is used. This means that common intelligence gathering tactics, such as port scans, won't reveal the existence of the VPN. From the perspective of an attacker, the port being used by Wireguard appears to be closed. This can help prevent attacks, including DOS, because an attacker simply won't know that there's anything there to attack.

Now, when I set about trying to install and configure Wireguard, I found that many of the online guides were either overly complex, or so incredibly simple that they only provided a list of commands to run without ever explaining what those commands did. The overly complex explanation turned out to be too confusing for me, but on the other hand, I'm also not someone to run random commands on my system without understanding what they do. I did eventually figure it out, though, so I thought I'd try writing my own explanation to hopefully offer others a middle ground.

Introduction

Wireguard uses public key encryption to create a secure tunnel between two "peer" machines at the network layer. It's free, open source software, and is designed to be simple to use while still providing a minimum attack surface.

In essence, wireguard creates a new ipv4 network over an existing network. On a device running wireguard, the wireguard network simply appears to be an additional network interface with a new IP address. On a typical laptop, for example, one might have several network interfaces, like lo, eth0 and wlan0, with an additional interface for wireguard appearing as wg0.

Installation

Installing Wireguard is typically done through your package manager, i.e. apt, dnf, pacman, etc, or optionally installed via a container package, via podman, flatpack, snap, appimage, or docker.

For example, apt install wireguard-tools. Depending on your system setup, you may need to preface this and some of the other commands in this walkthrough with the word "sudo".

Configuration

Once installed, wireguard will need to be configured. Wireguard considers devices to be peers, rather than using a client/server relationship. That means that there is no "primary" device that others connect to, rather, they can all connect to each other, just like a typical LAN. That said, for the sake of explanation, it is sometimes easier understand if we use the server/client labels, despite the devices not technically functioning in that relationship. In other words, I'm going to say "server" and "client" because I find it less confusing than saying "peer A" and "peer B".

We'll start on the "server" (aka. Peer A, not technically a server, yada yada) by running wg genkey | tee private.key | wg pubkey > public.key.
wg genkey produces a random "Private key", which will be used by the peer to decrypt the packets it receives. tee is a basic linux command that sends the output of one command to two destinations (like a T shaped pipe). In this case, it is writing the private key to a file called "private.key" and also sending the same data to the next command wg pubkey. This command creates a "public key" based on the "private key" it was given. Lastly, > is writing the output of wg pubkey to the file public.key. After running that command, we now have two files: private.key and public.key.

We can view the actual key by running cat private.key or cat public.key.

We'll repeat those same steps on the "client".

Next, let's decide what IP network address we want to use for the wireguard network. Remember that this needs to be different from your normal local network. There are several different network ranges available, but the most common is 192.168.1.0/24. Those first three "octets", 192, 168, and 1 define the network in this IP address, while the last one defines the device's address within that network. So, what's important here is that if your primary network is 192.168.1.0/24, that we assign the wireguard network a different address, like 192.168.2.0/24. By changing that 1 to a 2, we're addressing an entirely different network.

Now, we'll create the configuration files themselves. On the server, we'll create a configuration file by using a text editor, such as nano. nano /etc/wireguard/wg0.conf will create a file called wg0.conf and open it for editing. The name of this configuration will be the name of the network interface. It doesn't need to be wg0, but that's a simple, easy to remember option.

Here is an example configuration file for the server:

[Interface]
PrivateKey = (Your server's private key here)
Address = 192.168.2.10/24  
ListenPort = 51820

[Peer]
PublicKey = (Client's public key here)
AllowedIPs = 192.168.2.0/24
Endpoint = (see below)

PrivateKey is one that we generated on this server.

Address is the IP address that we're assigning to this server on the wireguard interface. This is our wireguard network address.

ListenPort is the "port" that wireguard is going to "listen" to. If you're not familiar with ports, think of it this way: an IP address is used to address packets to a specific device, and a port number is how you address those packets to a specific application on that device. In this case, Wireguard will be "listening" for packets that come in addressed to port number 51820. That number is somewhat arbitrary, but for uniformity, certain applications tend to use certain port numbers and 51820 is typical for wireguard.

The [Peer] section defines the other system/s on the network, in this case, our "client" system.

PublicKey is the public key created on the client device.

AllowedIPs tells wireguard which IP address it should forward to this peer. Here, we're telling it that any packets addressed to the 192.168.2.0/24 network should be forwarded out to this peer (our client system).

Endpoint is how we reach this peer. Because wireguard creates a network on top of an existing network, we need to tell wireguard how to get to the peer on the existing network before it can use the wireguard network. If you're connecting two devices on your LAN, we would just enter the normal IP address of the peer, such as 192.168.1.123. If you're connecting to another device over the internet, you will need to have a way to address packets to that device, such as a public IP address or a domain name. This will be specific to your network situation. Following the IP or domain name is a colon and the port number. For this example we'll assume the other machine is on your LAN at address 192.168.1.123 so we can just enter the IP address and the port number, like so: 192.168.1.123:51820. Lastly, save the file.

We'll then reverse this process on the 'client', assigning it a different IP address on the same network, such as 192.168.2.10/24 and using the clients PrivateKey and the server's Public Key. Once we've completed the configuration files on both device, we can use the command wg-quick up wg0, telling wireguard to start (bring up) the wg0 interface we just configured.

Final Notes

It should be noted that the "Endpoint" setting is only required on one of the two peers. This can be useful if you want to use wireguard while you're away from your home. My "server" at home may keep the same public IP all the time, but my portable laptop will have a different public IP address every time I move to a new network. In this case, I would remove the "endpoint" setting entirely from the configuration file on the server, and only use that setting on the laptop. Remember, the one system needs to be able to connect to the other system over an existing network before wireguard can create the VPN connection between them. Most of the issues a user might encounter while trying to set up wireguard isn't actually related to wireguard itself, but rather the underlying network. Firewalls and Network Address Translation (NAT) are the most common causes of problems, but the steps to address these issues will vary significantly depending on your network situation.

9 Dec 2024

My Proxmox Disaster Recovery Test (of the unplanned variety)

Welp, I lost another drive.

Homelab disasters are inevitable; expected even. Recently, a failed disk resulted in losing an entire node in my Proxmox cluster. While the initial shock was significant, a solid  backup/restore plan and a High Availability (HA) setup ensured a pretty swift recovery.

When I first started playing with my homelab, I was primarily using RPis and I lost a couple SD cards to corruption or failure. This helped to demonstrate to me the importance of regular backups, and I’ve made an effort to back up my homelab systems ever since. I now virtualize most of my servers via Proxmox and perform nightly backups to a local Proxmox Backup Server (PBS), which is then synchronized to an offsite PBS server. I’ve tested the restore function a few times and it seemed like a fairly straight-forward process.

For background, my current Proxmox cluster is comprised of three Lenovo Tiny SFF PCs. Two of those PCs are currently running only a single internal storage device which is used for both the boot and host partitions as well as storage for all of the guest OSes. This means that if a disk fails, it takes out the entire node and everything running on it.  

...Which is exactly what happened a couple weeks ago. Booting into BIOS showed that the drive failed so hard that the system didn’t even acknowledge there was a drive installed at all. The drive, by the way, was a Critical branded NVMe that I had purchased only two months prior. That’s just enough time to be outside of Amazon’s return period, yet significantly short of any reasonable life expectancy… but I digress. With the failing of the drive, I lost that node’s Proxmox host OS and all of the VMs and containers running on that node. For the HA enabled guests, they were automatically migrated to one of the remaining nodes, exactly as intended (yay!). For the non-HA guests, I had to manually restore them from PBS backup. I was quite pleased with how quick and easy it was to restore a guest from PBS. Everything could be done through the web gui in just a couple clicks. It’s obviously never fun to lose a disk, but PBS made the recovery pretty painless overall.


With the guests back up and running again, I removed the failed node from the cluster and purged any its remaining config files from /etc/pve/nodes.

For the failed node itself, I had to replace the drive and then reinstall Proxmox from scratch. From there, I pointed apt to my aptcacher-ng server and then ran a quick Post Install Script, before configuring my network devices and finally adding the “new” node back into the cluster. The whole process took only a couple hours (including troubleshooting and physically installing the new drive), and most of the hosted systems (such as this blog) were only offline for a handful of minutes, thanks to the High Availability set-up.

Needless to say, I was quite happy with my PBS experience.   
...And not so much my experience with Critical’s NVMe drives. 

23 Nov 2024

Caching Apt with Apt-Cacher NG

It recently occurred to me that as I update each Linux container or VM, I'm downloading a lot of the same files over and over again.  While the downloads aren't huge, it still seems wasteful to request the same files from the repo mirrors so many times... So why not just download the update once and then distribute it locally to each of my systems?  

That's the purpose of a caching proxy.

I chose apt-cacher ng as it's very simple to setup and use, so I spun up a dedicated LXC and installed apt-cacher ng via apt. Once it was up and running, it was just a matter of following the included documentation to point all of my other systems to that cache.

After upgrading just a couple of systems, I can already see the cache doing it's job:

Those "hits" are requests that were able to be fulfilled locally from the cache instead of needing to download  the files from the repo again. Since this is caching every request, it actually becomes more efficient the more that it's used, so hopefully the efficiency will increase even more over time.

So what exactly is happening?

First, this is not a full mirror of the Debian repos. Rather, apt-cacher ng acts as a proxy and cache. When a local client system wants to perform an update, it requests the updated packages from apt-cacher instead of the Debian repo directly. If the updated package is available in apt-cacher's local cache already, it simply provides the package to the requesting client. If the package is not in the local cache, then the proxy requests the package from the repo, provides that package to the client, and then saves a copy of the package to the cache. Now it has a local copy in case another system requests the same package again.

Some packages, like Crowdsec, are only installed on a single machine on my network, so the cache won't provide a benefit there. However, since most of my systems are all running Debian, even through they may be running some different services,  they will still all request a lot of the same packages as each other every time they update, like openssh or Python.  These will only have to be downloaded the very first time they're requested, and all of the subsequent requests can be filled from the proxy's local cache.

Do you use a cache in your homelab? Let me know below!

Pixel 8 Pro Modem Issues

Not much of a post here, but I figured I share an issue I've encountered with my Google Pixel 8 Pro.

A few weeks ago, my Pixel phone started reporting "no connection" and prompting to "insert SIM card."  I first tired removing the SIM card and cleaning it with some IPA, and then doing some common "fixes", such as resetting the network settings and toggling airplane mode on and off.  I did find that rebooting the phone entirely would remedy the problem temporarily, but that it would return hours or even mere minutes later.

I then began trying more drastic measures... I first requested an eSIM from my carrier. I thought that perhaps the SIM reader may have been physically damaged in some way, and thought that an eSIM might be the solution. Bizarrely, my carrier insisted that only iPhones support eSIMs and refused to provide me with an eSIM code to use, even after I explained that the Pixel phones do, in fact, support eSIMs (and have supported them even before Apple hardware did).  I was considering changing carriers, anyway, so I figured this was a good time to finally switch.  I signed up with a new carrier and received my new eSIM code right away.

I was expecting that I had finally found the solution to the problem, and was quite disappointed when a few hours after switching, I was greeted by the same "insert SIM" messages and lack of service.  

Now that it was clearly not a SIM issue, I started digging a little deeper. Enabling ADB, I was able to pull system logs from the phone via logcat. I found that there appeared to be a number of errors surrounding the cellular modem, which I suspect are responsible for the sudden connectivity issues. 

Here are the logs:

11-19 06:52:23.274 24777 24784 I modem_svc: State Monitor: Modem state changed from BOOTING to OFFLINE
11-19 06:52:23.287   949  1017 I modem_ml_svc: Poll timeout. interval=3000
11-19 06:52:23.288   949  1017 I modem_ml_svc: Modem state changed from 3 to 0
11-19 06:52:26.278 24777 24784 I modem_svc: Poll timeout. interval=3000
11-19 06:52:26.278 24777 24784 I modem_svc: State Monitor: Modem state changed from OFFLINE to BOOTING
11-19 06:52:26.290   949  1017 I modem_ml_svc: Poll timeout. interval=3000
11-19 06:52:26.290   949  1017 I modem_ml_svc: Modem state changed from 0 to 3
11-19 06:52:29.282 24777 24784 I modem_svc: Poll timeout. interval=500
11-19 06:52:29.294   949  1017 I modem_ml_svc: Poll timeout. interval=3000
11-19 06:52:32.288 24777 24784 I modem_svc: Poll timeout. interval=500
11-19 06:52:32.298   949  1017 I modem_ml_svc: Poll timeout. interval=3000
11-19 06:52:34.816  1089  1089 I cbd     : Picking unzipped modem image from /mnt/vendor/modem_img/images/default//modem.bin
11-19 06:52:34.816  1089  1089 I cbd     : Binary type specified path /mnt/vendor/modem_img/images/default//modem.bin is selected
11-19 06:52:34.811     1     1 W /system/bin/init: type=1107 audit(0.0:10975): uid=0 auid=4294967295 ses=4294967295 subj=u:r:init:s0 msg='avc:  denied  { set } for property=telephony.ril.modem_bin_status pid=425 uid=0 gid=0 scontext=u:r:vendor_init:s0 tcontext=u:object_r:default_prop:s0 tclass=property_service permissive=0' bug=b/315104235
11-19 06:52:34.819  1089  1089 I cbd     : BIN(/mnt/vendor/modem_img/images/default//modem.bin) opened (fd 0)
11-19 06:52:34.819   425   425 W libc    : Unable to set property "telephony.ril.modem_bin_status" to "4": PROP_ERROR_PERMISSION_DENIED (0x18)
11-19 06:52:34.820  1089  1089 I cbd     : CP binary file = /mnt/vendor/modem_img/images/default//modem.bin
11-19 06:52:34.820  1089  1089 I cbd     : CP REPLAY file = /mnt/vendor/modem_userdata/replay_region.bin
11-19 06:52:34.821  1089  1089 I cbd     : BIN(/mnt/vendor/modem_img/images/default//modem.bin) opened (fd 4)
11-19 06:52:34.823     1     1 W /system/bin/init: type=1107 audit(0.0:10976): uid=0 auid=4294967295 ses=4294967295 subj=u:r:init:s0 msg='avc:  denied  { set } for property=telephony.ril.modem_bin_status pid=425 uid=0 gid=0 scontext=u:r:vendor_init:s0 tcontext=u:object_r:default_prop:s0 tclass=property_service permissive=0' bug=b/315104235
11-19 06:52:34.827   425   425 W libc    : Unable to set property "telephony.ril.modem_bin_status" to "4": PROP_ERROR_PERMISSION_DENIED (0x18)

What you see above just kept repeating.

I don't know very much about Android specifically, but I don't think I should be seeing system processes returning permission errors.  I recall from back in the Nexus days, that the radio images usually had their own separate partition, so I briefly looked into whether I could reflash the radio partition, but I didn't see any factory images available. Perhaps that's not how the devices function any more.

In any case, I ultimately decided that it might just be easier to contact Pixel Support, since my phone is still under warranty.
I chatted with a support agent for a couple minutes and explained my problem and the troubleshooting steps I had tried. Not long after, I had a replacement unit ordered. I received confirmation from FedEx the next day that I had an overnight delivery inbound.  

This isn't the first time that I've had unsolvable issues with a Google phone. In fact, my Nexus 6P, Pixel 7 Pro and my current Pixel 8 Pro will have all been replaced at least once now. The only Google phone that never had an issue was my Pixel 5, which ironically I believe was also the cheapest. That said, Pixel Support has generally been helpful and willing to send a replacement device without much fuss. Their Advanced Exchange program even allows several weeks to transfer things to the replacement device before the old device needs to be sent back. 

I really wish that Essential had made a sequel to the PH1, as that was an excellent phone with a very clean version of Android. In some ways, the PinePhone feels like a bit of a spiritual successor, though, particularly with its expandability potential.   Maybe it'll finally be a viable alternative to the Pixel line for someone who refuses to touch another Samsung phone.     

17 Nov 2024

Cities Skylines 2 - Skyve Install in Linux

Ah, Cities Skylines 2:

I recently got back into Cities Skylines 2 after leaving the game for a while due to the release of PDX Mods essentially breaking all of the mods I had been using via r2modman at the time.

Now  that some time has passed, I decided to give it another go. I was quite interested to see that the Skyve mod manager has now come to C:S2, meaning that I didn't actually have to directly interact with PDX mods and could use a proper mod manager instead. Installing Skyve is supposed to be a two-step process, first you install the Skyve "mod", which is essentially just an installer, then you use that installer to install the actual Skyve program. Once installed, Skyve is a free-standing application that interacts with the C:S2 data without requiring steam or C:S2 to be running at the time. Not only can Skyve install and uninstall mods without needing to launch C:S2, but it also alerts you when other users have flagged a mod as broken or incompatible with another mod you have installed. 

Unfortunately, while C:S2 runs beautifully on Linux without any additional configuration (likely better than it does on Windows), Skyve was a different story. Skyve requires the Microsoft dotnet framework and doesn't appear compatible the opensource alternative Mono, which is commonly what would be used on Linux. It took a bit of trial and error to get Skyve running, so I thought I would share the process which ultimately worked for me:

The first step is simply installing the Skyve "mod" via PDX Mods. After installing the Skyve mod and restarting C:S2 twice, the "Install Skyve" button appeared in the menu as it was supposed to and clicking on it did bring up an installer interface. The installer appeared to run, but ended with an error message.  I switched from Proton Experimental to Proton GE 9.20, using ProtonUP to download and active the new Proton version. ProtonUp isn't needed, but it does make the process very simple.

After switching, the installer ran without any errors, but Skyve itself would not start.

Next, I used ProtonTricks to install the .Net 4.8 framework:

Open ProtonTricks and select the game:

Then select the "default" Wine prefix:

Then install component:

Then select .Net 4.8 (other 4.x versions might work):

That installed .Net, but when I tried to launch Skyve, I received an error about the .Net "RootInstall" registry not being found, so my next step was to install that:

In ProtonTricks, select "Run regedit"

Once in regedit, I navigated to

HKEY_LOCAL_MACHINE/Software/Microsoft/.NETFramework

 

There, I created the missing registry key, pointing to the .Net framework path:

 

Finally, my last step was to run Skyve.
I ran the binary via ProtonTricks' application launcher, which ensures that the program is run in the correct prefix. Skyve started and immediately recognized the C:S2 install and correctly listed all of my current mods and even suggested a few I should remove. After confirming that I wanted to remove the mod, I booted up C:S2 and found that the mod had been successfully removed. I tried installing a new mod as well, and that worked exactly as intended.

Hopefully this might be helpful to someone else who finds themselves struggling to get Skyve running.

Thanks for reading!

 

1 Nov 2024

New Router: BananaPi R3 - Part 3 - Configuration

After being subjected to numerous mean glares from the wife and accusations of "breaking the internet", I think I've got it all configured now...

https://www.k3can.us/garfield.gif

Ironically, the more "exotic" configuration, like the multiple VPN and VLAN interfaces were pretty simple to set up and worked without much fuss. The part that had me pulling my hair and banging my head against the desk was just trying to get the wifi working on 2.4GHz and 5GHz at the same time... Something wireless routers have been doing flawlessly for over a decade. After enough troubleshooting, googling, and experimenting, though, I had a working router.

I installed AdGuardHome for dns and ad blocking, but kept dnsmasq for DHCP and local rDNS/PRT requests. dsnmasq's DHCP options directs all clients to AGH, and AGH fowards any PTR requests it recessives to dsnmasq.

Next, I installed a Crowdsec bouncer and linked it to my Local Crowdsec Engine. Now, when a scenario is triggered, instead of banning the offending address at the server, it will be blocked at the edge router level instead. 

 

Lastly I installed and configured SQM (Smart Queue Management), which controls the flow of traffic through the router to the WAN interface. Without this management, the WAN interface buffer can get "bogged down" with heavy traffic loads and cause other devices to experience high latency or even lose their connection entirely. SQM performs automatic network scheduling, active queue management, traffic shaping, rate limiting, and QoS prioritization.

For a comparison, I used waveform to test latency under load.

Before SQM:

====== RESULTS SUMMARY ====== 	
Bufferbloat Grade	C
	
====== RESULTS SUMMARY ====== 	
Mean Unloaded Latency (ms)	51.34
Increase In Mean Latency During Download Test (ms)	76.01
Increase In Mean During Upload Test (ms)	8.69

After SQM:

====== RESULTS SUMMARY ====== 	
Bufferbloat Grade	A
	
====== RESULTS SUMMARY ====== 	
Mean Unloaded Latency (ms)	38.92
Increase In Mean Latency During Download Test (ms)	12.75
Increase In Mean During Upload Test (ms)	1.5

I have to say, I'm pretty happy with the results!
Going from a grade C with a 76 ms increase to a grade A with only a 12.75ms is a pretty substantial difference. This does increase the load on the CPU, but with the BPI R3s quad core processor, I expect that I'll still have plenty of overhead.

Overall, I think I'm happy with the configuration and the BPI R3 itself.

27 Oct 2024

Troubleshooting a minor network issue.

So, while surfing the web and reading up on how I wanted to configure my new BananaPi R3, I encountered a sudden issue with my network connection:

It seemed like I could reach the internet, but I suddenly lost access to my home network services. I tried to SSH into my webserver and was met with a "Permission denied" error.  Since I had earlier attached an additional USB NIC to connect to the BPI, I thought that perhaps the laptop had gotten the interfaces confused and was no longer tagging packets with the correct VLAN ID for my home network. Most of my servers are configured to refuse any requests that don't come from the management VLAN, so this explanation made sense. After poking around in the network settings, all of the VLAN settings appeared correct, but I did notice that the link was negotiated at 100 mbs, instead of the usual 1000. I tired to reconfigure my network settings, manually setting the link to 1000 mbs, resetting interfaces, changing network priorities, etc.  I then tired the classic "reboot and pray" technique, only to find that my wired network connect was down entirely. I wasn't receiving an IP from the DHCP server and the laptop kept reporting that the interface was UP, then DOWN, then UP again, then DOWN again.

Now I started to think that perhaps the issue was hardware related. My usual NIC is built into the laptop's dock, so I thought I would try power cycling the dock itself. This didn't seem to have any effect, besides screwing up my monitors placement and orientation. My next thought was that there might be an issue with the patch cable. "Fast Ethernet" (100mbs) can theoretically function on a damaged cable, so that might explain the lower link speed, and if the damage is causing an intermittent issue, that could also explain the up/down/up/down behavior.  

Being the smart homelabber that I am, I disconnected both sides of the cable and connected my ethernet cable tester. All 8 wires showed proper continuity, though, suggesting that the cable was fine.  When I plugged the cable back in, however, I noticed that I was suddenly getting the normal 1gbe again, but the link was still going up and down. This lead me to the conclusion that the cable likely was the issue, despite it passing the cable test.  I tried replacing the cable entirely with a different one, and found that I now had a stable, 1gbe connection, an IP addresses, and I could now access my network like usual.

Looking back, I think replacing the cable should have been troubleshooting step 1 or 2.

Also, in retrospect, there were some clues that might have let me fix the issue before this point, if I had only put the pieces together. I had noticed another day that the link speed had dropped to 100mbs, but it seemed to correct itself, so I ignored it instead of investigating. While working, I found that Zoom and other applications had started to report my internet connection as "unstable" and that a "speedtest" showed that my internet bandwidth to be drastically lower than it used to be. I assumed this was due to my ISP just being unreliable, since reduced speeds and entire outages are not unusual where I live. 

In hindsight, I think these were all indications that there was a level 1 issue in my network. In the future, I'll have to remember to not over-think things, and maybe just try the simplest solutions first.

 

 

New Router: BananaPi R3 - Part 2 - Flashing

Part 1 is here.

Now that the router is assembled, the next step is to decide where to flash the firmware. As I mentioned in the last post, this device offers a handful of options. Firmware can be flashed to the NOR, NAND, eMMC, or simply run from the SD card. From what I've read, it's not possible to boot from an m.2 card, though. That option is only for mass storage.

After a bit of reading, my decision was ultimately to install to all four!  Sort of...

Image showing the leads connected the UART connector and the DIP switches
The DIP switches and the leads connecting to the UART

My plan is to install a "clean" OpenWRT image to both the NOR and NAND.  The NAND image will be fully configured into a working image, and then copied to the eMMC. The eMMC will then be the primary boot device going forward.  If there's a problem with the primary image in the future, I would then have a cascading series of recovery images available. At the flip of a switch, I can revert to the known working image in the NAND, and if that fails, then I can fallback to the perfectly clean image in the NOR.

..And I do mean "at the flip of a switch". Due to the way that the bootable storage options are connected, only 2 of the 4 can be accessed at a time. Switching between NOR/NAND and SD/eMMC requires powering off the BPI and toggling a series of 4 dip switches, as seen on the official graphic below:

https://wiki.banana-pi.org/images/thumb/4/4c/BPI-R3-Jumper.png/320x107x320px-BPI-R3-Jumper.png.pagespeed.ic.Qyd9EK01n9.png

Switches A and B determine which device the BPI will attempt to boot from. Switch C sets whether the NOR or NAND will be connected and switch D does  the same for the SD and eMMC. To copy a image from the SD card to the NOR, for example, switch D must be high (1) to access the SD card, and switch C must be low (0) to access the NOR.  Since switches A and B set the boot device independent of which devices are actually connected, it would seem that you could set them to an impossible state to render the device unbootable, like 1110, or 1001. 

To accomplish my desired install state, I had to first write the OpenWRT image to the SD card on a PC, and then insert it into the RPI. With the the switches to 1101, I could write the image from the SD card to the NOR, then flip the switches (with the BPI powered off) to 1111 to copy the image to the NAND. Lastly, I can remove the SD card and reboot with the switches in 1010 to boot from the NAND. Then I'll configure the BPI into my fully configured state. This is the step I'm currently working on. I have it about 80% configured, but will need to actually install it in my network before I can complete and test the remaining 20%.  Once it is installed, tested, and fully configured, I'll  copy the NAND to eMMC, before finally setting the switches to 0110 and booting from the eMMC for ongoing use.

Unfortunately, I haven't had a good opportunity to take my network offline to install the new router, so the last bit of configuration might need to wait a little while...

 

22 Oct 2024

New Router: BananaPi R3 - Part 1 - Hardware

I've been using a consumer router from from 2016 (with OpenWRT hacked onto it) all the way here in 2024, and felt that it might finally be time for an upgrade. I settled on a BananaPi R3 because it was a reasonable price and seemed like it would be a fun project.

Here's the bare board as received:

You can see most of the physical features in this photo, including a USB3 port, two SFP ports, 5 RJ45 ports, an m.2 slot for a cellular modem, and a 26-pin GPIO header. On the bottom, there's also a m.2 slot intended for nvme storage, as well as slots for a micro SD and a micro SIM.  The CPU is a quad-core ARM chip paired with 2gb of RAM, and there's a handful of flash chips, providing NAND, NOR and eMMC.   Quite a lot of options!

My plan is to install OpenWRT to the NAND storage. I suspect the nvme might be useful if I wanted to run a small file server or something, but that's not in the plan for now.

 

The first step I took in assembly was to apply some thermal pads to the chips and then attach a cooler and fan.

The thermal pads are "OwlTree" brand, but I don't have any specific preference to them, I just happen to already have them on-hand from a previous project. The CPU is a 0.5mm pad applied, and I applied 1.5mm pads to the remaining chips.

Thermal pad applied to CPU

After applying  pads to all of the chips, I attached the cooler and plugged in the fan.

The next step was to install the board into the case. I went with the official BPI-R3 case. The quality is surprisingly nice and looks great once assembled. After installing the board I then installed the pigtails for the eight (yes, eight) antennas and applied some basic cable management.

Board installed into case and coax attached and routed to antenna ports.  

Now, I can't finish putting the case together quite yet, since I'll need access to the UART pins to install Openwrt to the NAND flash. The  UART header can be seen on the right side of this photo, but there is no way to access it once the case is assembled.

But, that's enough for today. I'll post an update once I make some progress towards getting OpenWRT flashed.

20 Oct 2024

Wavelog

I posted a while back about online amateur radio logging (here) and wanted to do a follow up on some of the client software I've been using to log contacts and sync to both LOTW and QRZ.

I used to use CQRLOG, which is still a great logger, but it is primarily intended to be installed directly on the PC you're logging from. If you want to view QSOs or add a new entry, you need to have access to that PC.
For greater flexibility, there are now self-hosted server-based logging platforms which allow access from any device with a web browser and internet access, such as Wavelog and Cloudlog. I'm going to refer to Wavelog throughout this post, but the features are similar between them both Wavelog and Cloudlog (the former being a fork of the latter). 


Wavelog is a PHP application intended for a LAMP stack, although it actually works on a variety of systems: more on that at the end.

To start, here's a quick screenshot of the QSO entry, on both desktop and mobile:

At a glance, you can see all of the details of the station as soon as you enter the callsign. If the station has a QRZ profile, it will even display their profile photo!

Once you start building your logs, Wavelog can create a number of reports and analytics based on your logbooks, such as:

Or band usage:

It can also track progress towards awards, such as:

 

One of the features that I particularly enjoy is that you can upload QSL cards into your logbook, attached to the QSO entry: 

It can also automatically retrieve eQSL cards, as well!  As I mentioned before, Wavelog can sync entries to LOTW, QRZ, eQSL and more, and will show you the confirmation status for each.

If you happen to use Gridtracker, it can log directly to Wavelog. There is also a small helper application which will allow you to log directly from wsjtx and automatically sync your qso entry page to your radio via FLRig (freq, band, etc).

 

I've been running a Cloudlog server for both myself and my father until about a month or two ago when I switched to Wavelog. The migration from one to the other was fairly straight forward, and the Wavelog wiki provides instructions.  I'm running Wavelog on an Nginx webserver with a MariaDB backend, but there is also a Docker image if you want something simpler.  I did test Cloudlog on a lighttpd server, as well, and encountered no problems.  

So far, I'm happy to report that Wavelog has been quite reliable. The only downtime I've encountered is when I've accidentally broken the webserver itself, and that's certainly not the fault of Wavelog.

If you're looking to try a new logging solution, Wavelog would get a strong recommendation from me.

Check it out at Wavelog.org

 

14 Oct 2022

Steamdeck + Ham radio = SteamedHamDeck?

- page 1 of 2