Category: Networking

blacklist_secint_eventsLast week I had a situation where an external client was reporting a complete inability to access any of our internet-accessible resources. If they changed IPs, though, they were fine. Considering it was over the Labor Day weekend, that we hadn’t changed *anything*, and that only this one client was having the problem, it seemed clear that the cause wasn’t on my side. They were small with outsourced IT, though, so pointing the finger back didn’t help much.

The fly in the ointment even for pointing at a cause was also that I didn’t have any logs of their traffic getting dropped. Normally I can search for initiating IPs or zoom into a small time window and find connections. Not this time. That left me with my own mental loophole of doubt about whether I actually did have an issue on my end.

So I opened a case with Cisco TAC, gave them the troubleshoot file dumps and asked for analysis. They stumbled some because I wasn’t able to attempt a reproduction on demand, since I didn’t have access to the client’s network to try things. TAC tried disabling some completely unrelated Snort rules, which did nothing (as expected). Eventually, I added a temporary explicit trust rule on their traffic so that they could regain access while we dug deeper. Once allowed, the connections were visible in FireSight Management Center (Defense Center) > Analysis > Connections > Events.

A couple days later, TAC returned with the root cause. The Sourcefire Security Intelligence Feed for Malware was blocking the client’s IP. That would have been really great to know on day 1, so I could have asked the client’s IT to address it. Of course, when I did give that root cause back to their IT, they responded that they knew about being blacklisted two days before reporting access issues to our network. Did they mention that at all? Nope. Where’s the love, people?

Networking Security Technology

We’re in the process of updating our load balancing platforms and are migrating several test/dev and backoffice applications from Kemp Virtual LoadMaster (VLM) load balancers to F5 BIG-IP Local Traffic Manager (LTM) virtual edition (VE). Wow…three abbreviations in the first sentence. Buckle up :).

One of the services that we migrated this week was the Azure Rights Management Server (RMS) Connector. If you’re here, you probably know what the Azure RMS connector is, but just in case, here’s the short definition from Microsoft:

The Microsoft Rights Management (RMS) connector lets you quickly enable existing on-premises servers to use their Information Rights Management (IRM) functionality with the cloud-based Microsoft Rights Management services.

This seemed like it would be a simple migration when I started, but I couldn’t get the health monitor to report the servers up. It was all green on Kemp, but not on F5, with all the same check parameters. I tried it in a browser (IE, Chrome and Firefox) and that puzzled me more. The only way to load the check page (https://<azurermsconnector_fqdn/_wmcs/certification/servercertification.asmx) was to authenticate with domain user credentials. Strange. It seems that the VLM, at least in v7.1-20b, was accepting a 401 as a healthy response. But that didn’t help with BIG-IP…

BIG-IP (F5) has a great CLI utility that helped verify that authentication was indeed the hold up on its health monitor. Using openssl s_client, I checked using the monitor parameters and received a 401 unauthorized error:

 <h2>401 - Unauthorized: Access is denied due to invalid credentials.</h2>
 <h3>You do not have permission to view this directory or page using the credentials that you supplied.</h3>

I tried putting some domain credentials into the username and password fields on the BIG-IP health monitor, but that only succeeded in locking out the account. Then my colleague came across something that pointed us to use the account UPN ([email protected]) and that did the trick. Here’s the step-by-step from start to finish:

 1. Create HTTPS Health Monitor (Local Traffic > Monitors > Create…)

Fields below are required or otherwise different from default values

  • Name: rms_conn_https (feel free to name it according to your format)
  • Description: HTTPS monitor for Rights Management Server Connector
  • Type: HTTPS
  • Parent Monitor: https
  • Send String: GET /_wmcs/certification/servercertification.asmx HTTP/1.1\nHost: rmsconnector.domain.com\n
    (make sure you change the “Host:” to the FQDN of your RMS connector)
  • Receive String: ServerCertificationWebService
  • User Name: [email protected] (replace with your domain user service account)
  • Password: <password of above account>

f5_rms_monitor

Microsoft Networking Technology

firesightsplashI started to title this a “Review” of the Cisco ASA with FirePOWER, but my objective is to highlight a few limitations of the integrated solution so that potential customers understand the product. It may turn out to be a review after all, but that’s the focus.

Let’s set some product context. Cisco completed its acquisition of Sourcefire on October 7, 2013, and its initial integration into the Cisco Security family on November 10, 2014. That makes this union very fresh–think of Cisco FirePOWER as newlyweds. They’re starting to share the same roof, but carry a lot of individuality and his/her domain around with them.

Next, let’s zoom in on the word, “Services”, or as you may see elsewhere, “Module”. Sourcefire makes a number of standalone, independent intrusion prevention system and application firewall appliances (i.e. 7000 series, 8000 series). When Cisco and Sourcefire united, they introduced the ability to put a dependent Sourcefire module into the Cisco ASA 5500-x next-generation firewall family. One Cisco partner described it as functioning like a virtual machine within the ASA (of sorts). Summation: it needs the host (ASA) to survive.

This “Module” should actually be packaged and marketed as a “Starter Kit” or an entry-level, feature-limited offering (with no building-block upgrade path; it’s a hardware ceiling). And perhaps it is by some Cisco VARs, but it’s new, so I think many are still coming up to speed with what it brings to the table.

To justify my above assertion, I’ll highlight four characteristics that have affected or disappointed me in my deployment, and that have motivated a new set of quotes to move to the hardware/standalone solution.

Networking Security Technology

We’ve been setting up Active Directory Federation Services (ADFS) on Windows Server 2012 R2 to tie up with Office365, and we ran into a snag with load balancing ADFS on our aging F5 BIG-IP LTM. It’s on the dinosaur end of the historical timeline, or to put it another way, “it’s in its sunset year”, and the latest supported code is 10.2.4.

This poses a bit of an issue with monitoring the ADFS servers, since the version shipping with Windows Server 2012 R2 includes a new SSL TLS feature called “Server Name Indication”, or SNI. The prehistoric 10.2.4 BIG-IP code doesn’t support SNI. Thankfully, Microsoft provides a way to monitor the servers over HTTP (instead of HTTPS), but the documentation we found–links below–was lacking an important detail.

Microsoft Networking Technology

We received an update from our Dell team today. It looks like politics of who’s the root cause are going to make all of us suffer for another 6 months at least. Read on…

3Sept2014 BG – We investigated this issue. Looks like it is due to anomalous behavior of HyperV NDIS stack. Driver allocates MSIX interrupts for VMQs at load time and sets interrupt affinity bases on the RSS processor set returned by NdisGetRssProcessorInformation call as per MSDN documentation. None of the host CPU is in the list of RSS processors below the base RSS Processor number RssBaseProcNumber.

Later on NDIS specifies cpu0 when it send OID to allocate VMQ. Driver doesn’t find any MSIX interrupt to satisfy the VMQ allocate OID and hence driver fails the OID.

Microsoft Networking Technology Virtualization

"Because ninjas are too busy"

I first ran into Splunk at VMworld 2012 where I picked up one of my favorite swag t-shirts, but I never took more than the t-shirt for a spin. Then last week my SolarWinds Kiwi Syslog maintenance reminder popped up. Kiwi is cheap, but it’s also cheap, if you know what I mean. It’ll grab those logs all day long, but translating data into useful info or simply searching it quickly just isn’t its strong suit. So I took a moment to search for leading syslog solutions that are Windows friendly.

Splunk Enterprise was near the top of the search results, and a quick perusal showed that it might be worth the time to trial.

Networking Technology

We’ve confirmed with Dell Support that QLogic has identified a bug with the QLE82xx network driver (at least through version 5.3.12.0925) and Virtual Machine Queues (VMQ). As of August 15, 2014, QLogic reports that they have reproduced the issue, but have not resolved it.

We have a case open with Microsoft Hyper-V Support and that case number has been shared with Dell and QLogic to coordinate troubleshooting and support as the issue becomes more visible in the community. I’ll post updates as we have them. There is word of a beta driver at some point, which we’ve expressed interest in testing.

Networking Technology Virtualization

In Part 1, I laid out a brief summary of VMQ and an example of the configuration that is appropriate for our four-socket, ten-core Hyper-V host. Here in Part 2, I’ll unpack the issue we’re facing in spite of our textbook configuration.

Following the guidance in VMQ Deep Dive, Part 2, using the commands in Part 1 of this blog, we find the below queue list. The three line items are default queues given to the Hyper-V host (HV01) on its physical ports and the related logical switch. We should be seeing the VMs on this host in that list as well, but we aren’t.

vmq_ps_not_zero

The Windows System event log shows the following error, which seems to be the issue with queuing.

vmq_oid_failed

Microsoft Networking Technology Virtualization

Last week I encountered a briefly puzzling situation that’s worth noting as a tip when replacing a server on the network and needing to keep the same hostname. We’re a…

Microsoft Networking Technology

Microsoft has been gaining ground in the virtualization sphere one step at a time since Hyper-V first premiered. While the some increments were negligible (or merely painstakingly obvious), they achieved significant breakthroughs in late 2013 with the release of all things “2012 R2”. The puzzle piece on which we’ll focus here is VMQ (specifically dynamic VMQ, or dVMQ).

get-netadaptervmqVMQ gives Hyper-V and System Center Virtual Machine Manager (VMM) Logical Switches what Receive Side Scaling (RSS) provides to physical servers; namely, it leverages multiple compute cores/interrupts to increase network traffic efficiency. The network teaming (or Load-Balancing Fail-Over, LBFO) configuration is important here, because it affects how VMQ maps queues to processors. The full table of possibilities is given halfway down the page of TechNet’s VMQ Deep Dive, Part 2. In a nutshell, some configurations need NIC queues to overlap the same processors (so that all queues are everywhere), while others need segregation (so every queue has its own unique core).

Microsoft Networking Technology Virtualization