Last week I had a situation where an external client was reporting a complete inability to access any of our internet-accessible resources. If they changed IPs, though, they were fine. Considering it was over the Labor Day weekend, that we hadn’t changed *anything*, and that only this one client was having the problem, it seemed clear that the cause wasn’t on my side. They were small with outsourced IT, though, so pointing the finger back didn’t help much.
The fly in the ointment even for pointing at a cause was also that I didn’t have any logs of their traffic getting dropped. Normally I can search for initiating IPs or zoom into a small time window and find connections. Not this time. That left me with my own mental loophole of doubt about whether I actually did have an issue on my end.
So I opened a case with Cisco TAC, gave them the troubleshoot file dumps and asked for analysis. They stumbled some because I wasn’t able to attempt a reproduction on demand, since I didn’t have access to the client’s network to try things. TAC tried disabling some completely unrelated Snort rules, which did nothing (as expected). Eventually, I added a temporary explicit trust rule on their traffic so that they could regain access while we dug deeper. Once allowed, the connections were visible in FireSight Management Center (Defense Center) > Analysis > Connections > Events.
A couple days later, TAC returned with the root cause. The Sourcefire Security Intelligence Feed for Malware was blocking the client’s IP. That would have been really great to know on day 1, so I could have asked the client’s IT to address it. Of course, when I did give that root cause back to their IT, they responded that they knew about being blacklisted two days before reporting access issues to our network. Did they mention that at all? Nope. Where’s the love, people?
