After watching the SpaceX Falcon9 CRS-24 Launch (which was fantastic, albeit cloudy and rainy) at Space View Park in Titusville, Florida, I went to a highly rated bakery and coffee shop in town called Sunrise Bread Company. I got setup and was ready to dive into work when I found myself becoming quickly frustrated by networking issues. In particular, websites wouldn’t load due to what appeared to be networking issues. I could ping CloudFront’s DNS servers and make DNS queries without any issue. I could also establish VPN connections with ProtonVPN, but no traffic would pass.
Here were the results for Safari and Chrome:
The issue was intermittent; I could load Google search but not the National Park Service website. This led me to think there was a DNS caching problem. Due to weird DNS caching issues on macOS and Apple’s new iCloud Private Relay, I automatically thought those were the cause. The first thing I did was kill the DNS responder service, which normally resolves macOS DNS resolution issues.
$ sudo killall -HUP mDNSResponder
That did not make any difference this time. Then I disabled iCloud Private Relay for the wireless connection.
Still no difference, but I have no confidence in that checkbox. So I disabled it system wide (
System Preferences > Apple ID > Uncheck Private Relay (Beta)). Nada.
Everything seems to be fine… Having a headache I just restarted my mac. Again, no difference. I checked Apple’s Status Page to see if iCloud Relay was functioning properly, just in case local settings weren’t taking effect (again, because I have no confidence in this new beta feature.) All systems nominal. What about AWS? They recently had a couple major outages, and the NPS website is hosted on CloudFront, which isn’t working (while Google Search is.) All green there, too. Time to pull out the big guns.
I opened Wireshark and started a WiFi Capture with VPN (using WireGuard). Okay, all local traffic. It all looks relatively fine with local patrons broadcasting their deepest secrets to anyone who will listen. Doing another capture on the VPN Tunnel interface, I saw that there were loads of DNS queries that were being continuously resent without response. Then I did a WiFi Capture without VPN. Wow. Look at all those TCP resets.
Turns out it’s not my mac after all. Sorry, Apple. I asked around the coffee shop and everyone was having intermittent connectivity. One regular patron mentioned that a while back someone was pirating content over the coffee shop network and the ISP has been throttling bandwidth ever since. Interesting. I remember that Comcast performed TCP Reset attacks on its own customers over protocols they deemed illegitimate (i.e. BitTorrent), so there is precedence. I did a quick ARIN lookup on the public IP address at the coffee shop and saw that Charter Communications (now known as Spectrum) is the ISP. But I was skeptical; why would Spectrum perform TCP resets on HTTPS connections to the National Park Service website or AWS CloudFront?
It’s got to be the router, I thought. HTTP to the default gateway…prompted for credentials…darn, default credentials were changed. That’s okay, it provides a handy password reset page that identifies itself: Netgear R7000. A Google search for “R7000 tcp resets” brought me to a Reddit post from ~170 days ago with several other people who have the same problem.
Then I wondered why VPN wasn’t working. I use ProtonVPN, which defaults to the WireGuard protocol. Wikipedia states that WireGuard uses UDP. I started changing protocols to troubleshoot.
- OpenVPN TCP - Does not work
- OpenVPN UDP - Works
- IKEv2 - Works
- WireGuard - Does not work
The router (or ISP) is definitely doing some sort of TCP packet inspection and forgery. But WireGuard uses UDP and unlike all the other UDP protocols I tested (DNS, OpenVPN UDP), traffic did not traverse the tunnel. Note that in all cases the tunnels established successfully. I don’t know enough about WireGuard to troubleshoot further without additional research. I can only speculate that the router (or ISP) may whitelist or blacklist certain UDP ports or that one of WireGuard’s protocols is being blocked or modified in some other way.