Sometimes I think I over-complicate things.
In fact, I know that I do. e.g. this blog. I really don't need this to be hosted in my homelab on a high-availability server cluster. I could just use Blogger or any other free, hosted blog solution.
Anyway, today's example is an oddity I noticed in my DNS logs. Well, actually, it started when I noticed that the RAM in my router was filled with files in /tmp. Like a whole GB of files. I traced it down to my DNS logs, which were occupying almost the entire /tmp mount. I found that I had apparently set the log retention to 90 days. I reduced it to something more reasonable and I think that should solve the problem. However, that also made me curious about what was in those logs...
I found that the majority of the DNS requests were for "0.10.0.5.1". Which sort of looks like an IP address, except for that extra 0 at the start.
Since it's not a valid address, something was interpreting it as a hostname and trying to resolve it. I knew from the logs that it was originating from my webserver (the same server which hosts this blog), but I didn't know which application was actually making the requests. I was able to catch the outgoing port with lsof
, but it was an ephemeral port and too short-lived for me to trace it back to an application.
I spent a while trying to catch it with auditctl rules, but I'm not very familiar with auditd and I wasn't making any progress. I google'd around and even tried a chat bot to come up with more ideas on how to catch this application; strace, bpftrace, and so on, without any success.
Finally, I decided to try doing it "the dumb way."
Instead of trying to outsmart this application and catch it in the act, I figured something must be configured to send requests to 0.10.0.5.86, so I just searched the entire /etc/ folder for that string via grep -Rnw '/etc/' -e '0\.10\.\0\.5\.1'
.
And grep: /etc/systemd/timesyncd.conf:20:NTP=0.10.0.5.1
Well, crap. I had configured timesyncd months ago to use my router as its time server and apparently I accidentally added an extra 0 to the IP address.
Oh well. A quick edit and systemctl restart systemd-timesyncd.service
later, and the issue is fixed.