Tag - nginx

Entries feed - Comments feed

19 Feb 2025

Inspect What You Expect

That's a phrase I heard for the first time when I became a manager; my boss would tell me I needed to "inspect what I expect." To be honest, I don't think I fully understood the phrase until a lot later. In the context of leadership, it means that if you have certain expectations for your team, you should actively verify that those expectations are being met and investigate the cause if they're not. Don't wait until eval season to say "You failed to meet expectations", instead, review expectations regularly, and if someone isn't meeting them, it's your job to figure out why. Have you defined the expectations in a way that they understand? Are your expectations realistic? Did you provide the resources they needed to meet the exceptions? Did you provide guidance when needed? And so on.

You might be asking: "Sure, but why is this on a blog about computers and stuff?"

Expectations


You might have the expectation that your network is secure. You don't remember adding any firewall rules or forwarding any ports, or maybe you followed some YouTube tutorial and they told you to proxy your incoming connections through Cloudflare. Maybe you set up a "deny all" line in that NGINX config or you only access your network through a VPN. You expect that your network is safe from outside actors.

To use a real world example, my father set up several Lorex brand CCTVs on his home network to save video to an NVR. He configured a VPN to allow secure remote access to the NVR and assumed that everything was safe. He didn't create any firewall rules or port forwarding to allow access by anything other than the VPN. He expected that this was safe, and in theory, it should have been.

Inspection


We traveled together for a holiday and while drinking coffee one morning, I showed him Censys. Censys is search engine, but unlike Google or Bing, which search for webpages, Censys searches for the actual devices. It indexes the IP addresses of everything on the internet and everything about them. You can write queries to search this index for almost anything you can think of.

For a quick demonstration, I just searched for his home IP address, and we were both surprised when we saw the results:
not actually his results

Umm... What?

For those who might not know RTSP stands for Real-Time Streaming Protocol. It is a protocol designed to stream video over a network, commonly used for CCTV cameras. The HTTP port was the HTML login screen for one of his cameras. We clicked it and I sarcastically tried a default login (admin, admin) and we found ourselves staring at his basement.
As far as he knew, he took all the appropriate steps to secure the devices on his network and expected that nothing was unsafely exposed, but he hadn't inspected that expectation. We found that the camera was factory configured to expose itself via uPnP, a technology which allows devices to request changes to port-forwarding and firewall rules without user involvement. This is supposed to allow for easy set-up by inexperienced users, but it can also create significantly compromise security without the use knowing about it. In our case, my father is not an inexperienced user, he's been a computer engineer since the 80s and has even worked for one of the major producers of networking equipment. He took all the right steps to get his expected result, he just hadn't inspected it ensure that his expectation was being met.

Inspect What You Expect


Censys can be a great starting point when evaluating your network by helping you to understand what you have exposed to the internet. Censys queries can be as simple as an IP address if you just want to see a single point, or they can search broadly for very specific things.

Here are the results of a very simple query looking for exposed FTP servers based out of japan:
japan

There's a lot of FTP servers, obviously, but did you notice that SSH server on port 10022? Some people expect that services will be hidden if they run them on non-standard ports, but don't inspect to see if that's actually the case. Here, we can see that the SSH server is still quite visible, despite being on a non-standard port, just like those non-standard HTTP servers on the other entries. Clicking into an entry will provide even more information, like the software versions, request responses, and so on.

Through Censys, I realized that I was running an older version of Nginx than I thought I was, and that this older version had a number of vulnerabilities that were patched in later versions. I expected that I was running the a current version, but my inspection showed me otherwise.

Final thoughts


While a tool like Censys isn't the only tool you should use to inspect your security expectations, it's a great starting point, since it can show you what your network looks like to the internet. t's also a fun tool to use to explore the internet from a different angle. Instead of just searching the surface of the web for youtube videos and news stories, try searching deeper for Roombas, smart lights, or security cameras.

The important takeaway, though, is that just because you think that something is working how you expect it to, doesn't always mean that it is.
The only way to know for sure is to inspect what you expect.

18 Jan 2025

Let's Gopher!

I've decided it's time to get my gopher server running again, and wanted to outline the basic steps for anyone else looking to dive into the gopherhole.

install

I personally like gophernicus, a "modern full-featured (and hopefully) secure gopher daemon." Assuming that you're running a Debian server, we can install a simple gopher server via apt:

apt install gophernicus

installing screenshot

This should also install a socket unit and create a service template for gophernicus.

config

Gophernicus defaults to serving from either /var/gopher/ or /srv/gopher. On my install, the default config file is stored at /etc/defaults/gophernicus. I feel like /etc/defaults/ isn't used very often these days, but the intention is to provide a simple way for a developer to provide a default configuration for their application. We can create a new config somewhere reasonable, like /etc/gophernicus or just modify the existing file in /etc/default/ like a lunatic. I'm choosing the latter, of course.

The important thing to add is a hostname, but we can also turn off any features we don't want.

My config looks like OPTIONS=-r /srv/gopher -h gopher.k3can.us -nu.

The -nu option just disables the "personal" gopherspace (serving of ~user directories). Unless you know what this does and intend to use it, I'd suggest disabling it.

testing

We should now be able to start the service and access our empty gopherhole via systemctl start gophernicus.socket, which will create an instance of the gophernicus@.service unit. We can run a quick test by creating a file to serve and viewing the gophermap. To create the test file: touch /srv/gopher/test.txt, and then we can fetch the gophermap via telnet [ip address]-70.

 Trying 192.168.1.5...
Connected to 192.168.1.5.
Escape character is '^]'.

i[/]    TITLE   null.host   1
i       null.host   1
0hello-world.txt                       2025-Jan-18 09:47     0.1 KB /test.txt   gopher.k3can.us 70
.
Connection closed by foreign host.

That little jumble of text is our gohpermap. Lines starting with i indicate "information", while the 0 indicates a link to a text file. Gophernicus creates this map automagically by examining the content of the directory (although it also provides the option of creating a map by hand). To add files and folders, we can simply copy them into the /srv/gopher and gophernicus will update the map to include the new files.

From here, if we want to expose this publicly, can simply route/port-forward through our router.

In my case, though, I'm going to need to configure a few more components before I open it up to the public... First, I use (apparmor)[https://apparmor.net/] to limit application access, and second, my webserver lives behind a reverse proxy.

apparmor

For apparmor, I created a profile for gophernicus:

include <tunables/global>
# AppArmor policy for gophernicus
# by k3can

/usr/sbin/gophernicus {
  include <abstractions/base>
  include <abstractions/hosts_access>
  network inet stream,

  /etc/ld.so.cache r,
  /srv/gopher/ r,
  /srv/gopher/** r,
  /usr/bin/dash mrix,

}

This profile limits which resources gopernicus has access to. While gophernicus should be fairly secure as is, this will prevent it from accessing anything it shouldn't on the off-chance that it somehow becomes compromised. Apparmor is linked above if you want to get into the details, but I'm essentially telling it that gophernicus is allowed to read a number of commonly-needed files, run dash, and access its tcp stream. Gophernicus will then be denied access to anything not explicitly allowed above.

nginx

Lastly, to forward gopher through my reverse proxy, I added this to my nginx configuration:

#Gopher Stream Proxy

 stream {
     upstream gopher {
         server 192.168.1.5:70;
     }

     server {
              listen 70;
              proxy_pass    gopher;
     }
 }

Since nginx is primarily designed to proxy http trafic, we have to use the stream module to forward the raw TCP stream to the upstream (host) server. It's worth noting that as a TCP stream, ngnix isn't performing any virtual host matching, it's simply passing anything that come into port 70 on to the uptream server. This means that while I've defined a specific subdomain for the host, any subdomain will actually work as long as it comes into port 70; gopher://blog.k3can.us, gopher://www.k3can.us, and even gopher://sdkfjhskjghsrkuhfsef.k3can.us should all drop you into the same gopherhole.

access

While telnet will show you the gophermap, the intended way to traverse gopher is through a proper client application. For Linux, Lynx is a command-line based web-browser/gopher client available in most repos, and for Android, DiggieDog is available through the PlayStore.

next steps

Now all that's for me to do is add content. I used to run a script that would download my Mastodon posts and save them to a "phlog" (a gopher blog), and I could potentially mirror this blog to there, as well. That helped me keep the content fresh without needing to manually add files. I haven't quite decided if I want the gopherhole to primarily be a mirror of my other content, or if I want to be more intentional with what I put there.

Besides figuring out content, I'm also curious about parsing my gophernicus logs through Crowdsec. Unsurprisingly, there's not currently a parser available on the Crowdsec Hub, so this might take a little tinkering...