That's a phrase I heard for the first time when I became a manager; my boss would tell me I needed to "inspect what I expect." To be honest, I don't think I fully understood the phrase until a lot later. In the context of leadership, it means that if you have certain expectations for your team, you should actively verify that those expectations are being met and investigate the cause if they're not. Don't wait until eval season to say "You failed to meet expectations", instead, review expectations regularly, and if someone isn't meeting them, it's your job to figure out why. Have you defined the expectations in a way that they understand? Are your expectations realistic? Did you provide the resources they needed to meet the exceptions? Did you provide guidance when needed? And so on.

You might be asking: "Sure, but why is this on a blog about computers and stuff?"

Expectations

You might have the expectation that your network is secure. You don't remember adding any firewall rules or forwarding any ports, or maybe you followed some YouTube tutorial and they told you to proxy your incoming connections through Cloudflare. Maybe you set up a "deny all" line in that NGINX config or you only access your network through a VPN. You expect that your network is safe from outside actors.

To use a real world example, my father set up several Lorex brand CCTVs on his home network to save video to an NVR. He configured a VPN to allow secure remote access to the NVR and assumed that everything was safe. He didn't create any firewall rules or port forwarding to allow access by anything other than the VPN. He expected that this was safe, and in theory, it should have been.

Inspection

We traveled together for a holiday and while drinking coffee one morning, I showed him Censys. Censys is search engine, but unlike Google or Bing, which search for webpages, Censys searches for the actual devices. It indexes the IP addresses of everything on the internet and everything about them. You can write queries to search this index for almost anything you can think of.

For a quick demonstration, I just searched for his home IP address, and we were both surprised when we saw the results:


Umm... What?

For those who might not know RTSP stands for Real-Time Streaming Protocol. It is a protocol designed to stream video over a network, commonly used for CCTV cameras. The HTTP port was the HTML login screen for one of his cameras. We clicked it and I sarcastically tried a default login (admin, admin) and we found ourselves staring at his basement.
As far as he knew, he took all the appropriate steps to secure the devices on his network and expected that nothing was unsafely exposed, but he hadn't inspected that expectation. We found that the camera was factory configured to expose itself via uPnP, a technology which allows devices to request changes to port-forwarding and firewall rules without user involvement. This is supposed to allow for easy set-up by inexperienced users, but it can also create significantly compromise security without the use knowing about it. In our case, my father is not an inexperienced user, he's been a computer engineer since the 80s and has even worked for one of the major producers of networking equipment. He took all the right steps to get his expected result, he just hadn't inspected it ensure that his expectation was being met.

Inspect What You Expect

Censys can be a great starting point when evaluating your network by helping you to understand what you have exposed to the internet. Censys queries can be as simple as an IP address if you just want to see a single point, or they can search broadly for very specific things.

Here are the results of a very simple query looking for exposed FTP servers based out of japan:


There's a lot of FTP servers, obviously, but did you notice that SSH server on port 10022? Some people expect that services will be hidden if they run them on non-standard ports, but don't inspect to see if that's actually the case. Here, we can see that the SSH server is still quite visible, despite being on a non-standard port, just like those non-standard HTTP servers on the other entries. Clicking into an entry will provide even more information, like the software versions, request responses, and so on.

Through Censys, I realized that I was running an older version of Nginx than I thought I was, and that this older version had a number of vulnerabilities that were patched in later versions. I expected that I was running the a current version, but my inspection showed me otherwise.

Final thoughts

While a tool like Censys isn't the only tool you should use to inspect your security expectations, it's a great starting point, since it can show you what your network looks like to the internet. t's also a fun tool to use to explore the internet from a different angle. Instead of just searching the surface of the web for youtube videos and news stories, try searching deeper for Roombas, smart lights, or security cameras.

The important takeaway, though, is that just because you think that something is working how you expect it to, doesn't always mean that it is.
The only way to know for sure is to inspect what you expect.

It recently occurred to me that as I update each Linux container or VM, I'm downloading a lot of the same files over and over again.  While the downloads aren't huge, it still seems wasteful to request the same files from the repo mirrors so many times... So why not just download the update once and then distribute it locally to each of my systems?  

That's the purpose of a caching proxy.

I chose apt-cacher ng as it's very simple to setup and use, so I spun up a dedicated LXC and installed apt-cacher ng via apt. Once it was up and running, it was just a matter of following the included documentation to point all of my other systems to that cache.

After upgrading just a couple of systems, I can already see the cache doing it's job:

Those "hits" are requests that were able to be fulfilled locally from the cache instead of needing to download  the files from the repo again. Since this is caching every request, it actually becomes more efficient the more that it's used, so hopefully the efficiency will increase even more over time.

So what exactly is happening?

First, this is not a full mirror of the Debian repos. Rather, apt-cacher ng acts as a proxy and cache. When a local client system wants to perform an update, it requests the updated packages from apt-cacher instead of the Debian repo directly. If the updated package is available in apt-cacher's local cache already, it simply provides the package to the requesting client. If the package is not in the local cache, then the proxy requests the package from the repo, provides that package to the client, and then saves a copy of the package to the cache. Now it has a local copy in case another system requests the same package again.

Some packages, like Crowdsec, are only installed on a single machine on my network, so the cache won't provide a benefit there. However, since most of my systems are all running Debian, even through they may be running some different services,  they will still all request a lot of the same packages as each other every time they update, like openssh or Python.  These will only have to be downloaded the very first time they're requested, and all of the subsequent requests can be filled from the proxy's local cache.

Do you use a cache in your homelab? Let me know below!