Last week I had a situation where an external client was reporting a complete inability to access any of our internet-accessible resources. If they changed IPs, though, they were fine. Considering it was over the Labor Day weekend, that we hadn’t changed *anything*, and that only this one client was having the problem, it seemed clear that the cause wasn’t on my side. They were small with outsourced IT, though, so pointing the finger back didn’t help much.
The fly in the ointment even for pointing at a cause was also that I didn’t have any logs of their traffic getting dropped. Normally I can search for initiating IPs or zoom into a small time window and find connections. Not this time. That left me with my own mental loophole of doubt about whether I actually did have an issue on my end.
So I opened a case with Cisco TAC, gave them the troubleshoot file dumps and asked for analysis. They stumbled some because I wasn’t able to attempt a reproduction on demand, since I didn’t have access to the client’s network to try things. TAC tried disabling some completely unrelated Snort rules, which did nothing (as expected). Eventually, I added a temporary explicit trust rule on their traffic so that they could regain access while we dug deeper. Once allowed, the connections were visible in FireSight Management Center (Defense Center) > Analysis > Connections > Events.
A couple days later, TAC returned with the root cause. The Sourcefire Security Intelligence Feed for Malware was blocking the client’s IP. That would have been really great to know on day 1, so I could have asked the client’s IT to address it. Of course, when I did give that root cause back to their IT, they responded that they knew about being blacklisted two days before reporting access issues to our network. Did they mention that at all? Nope. Where’s the love, people?
While the Googlesphere seems devoid of any ability to search/query Sourcefire’s blacklist feeds, it did render some Cisco documentation on enabling logging for blacklist traffic. This also would have saved me the entire TAC case, had I known it was disabled by default.
To enable logging for blacklists, follow these steps (based on version 5.4.1):
- Login to FireSight Management Center (formerly Defense Center)
- Go to Policies > Access Control
- Edit your active access control policy
- Select the third tab, Security Intelligence
- On the far right of the far right (fourth) column above the little Delete trashcans, click the Logging paper icon
- Check Log Connections and Defense Center (optionally other destinations)
- Click OK and then Save and Apply in the top right of the Access Control page
- Blacklist connections will now get logged