Network Analysis Unplugged

Practical Forensics - Part 2 of 2

This second of two parts on practical forensics illustrates a means for detecting a type of "technical" activity as mentioned in the CERT study: “Organizations failed to detect or ignored policy rules such as forbidden downloads”.  In the previous blog, I referred to this as red flag #5.  For illustrative purposes, we are going to check for a user searching for and downloading password cracking tools or illegal software key generator utilities.  If detected, there's a high probability that the user will actually use such a tool.

One of the things I love about advanced network analysis and forensics tools is the ability to easily apply custom triggers and triggers to find stuff inside packets.  Triggers are a special case of filters that allow us to “trigger” to start a packet capture and/or send an alert if we are capturing in real time.

In this case, we are going to take advantage of a special capability of the analyzer to search for an arbitrary word or pattern anywhere in a packet.  This “sliding pattern match” does not require any prior knowledge of where the pattern might be.  To optimize performance a bit, we are going to start at offset 54 inside the packet, which tells the analyzer to start at the beginning of the TCP payload for any upper layer protocol.  No sense in wasting CPU cycles looking for application data in the data link, IP, and TCP headers.

The screenshot you see here (from WildPackets OmniPeek) shows such a filter, using a combination of AND along with OR conditions.  We start out by looking for the words ‘password’ or ‘key’.  If password is found, it must match (AND) the words or patterns ‘crack’ OR ‘krack’ OR ‘recover’.  Thus phrases like password crack, password krack, password recover, password recovery (or the reverse order) will be found.  Likewise for ‘key serial’ <number>, ‘key generator’, ‘keygen’, and so on.  I’ve also told the analyzer to ignore case sensitivity.

Test it by searching on-line with your favorite search engine (which will trigger a hit right there) or going to any website containing such tools.  The filter and/or trigger will hit immediately – even if the tool is named something else, purveyors of such tools love to fill up their underlying web site code with key words to gain search engine positioning.

This is the tip of the iceberg when it comes to real time or forensics data mining.  Such tools can be invaluable in assisting you to effectively combat potential sabotage and espionage in your network.  For more ideas, check out the forensics filters available for download at the WildPackets Developer Network (Registration is free but a login is required).

Practical Forensics - Part 1 of 2

With the billions of dollars spent protecting our corporate networks from the outside world, when will we begin to pay serious attention to what happens inside our networks?  To continue the theme of Network Forensics from my previous entry, I'll take a closer at the security aspects with respect to your network from the inside.  We'll take a brief look at a Carnegie Mellon study, followed by a practical tip you can apply using your favorite forensics and analysis tool.

The world famous Carnegie Mellon Computer Emergency Response Team (CERT) published an interesting paper entitled “Comparing Insider IT Sabotage and Espionage:  A Model-Based Analysis."  This 108 page (gulp) study cannot be summarized in its entirety here, but if network security is of interest to you, I highly encourage downloading it.  The best part is that it was funded by the DoD and as such, is freely available.  Click here for a direct link to the paper.

The study examined “the psychological, technical, organizational, and contextual factors” which contributed to espionage and sabotage against IT.  Their research led to the following red flags:

1. Saboteurs had personal problems outside the workplace.
2. Stressful events such as internal reorganizations increase the likelihood of malicious acts.
3. Poor work ethics including performance or tardiness were often observed before and during sabotage.
4. Insiders had a tendency to “set-things up” such as creating back door accounts.
5. Organizations failed to detect or ignored policy rules such as forbidden downloads.
6. A lack of access control for both physical locations and on-line computing resources.

The report goes on to provide recommendations for further research to mitigate the risk.  One of the repeated themes is to acquire “improved data” related to things like interrelationships, stressful events, assess policy enforcement vs. technical rule violations, research tools for auditing and monitoring, etc.

While personal problems can be tough to deal with especially with touchy regulations like HIPPA, we can deploy tools to help us with technical matters.  The tool I have in mind provides powerful forensics capabilities as discussed in the previous blog.  Not discussed were specific best practices for using such a tool. 

To get your creative juices going, on Friday I’l share with you one such technique to help detect anomalous activity.  This technique detects scrupulous activity that you would otherwise not see with simple network statistics.

Stay tuned and check back!

Network Forensics

Quick, what’s the first word that comes to mind when you hear the word ‘forensics’?  CSI? Despite being an IT guy, I still have morbid thoughts along the lines of a pathologist slicing into a cadaver.  Or collecting stuff that might contain the perps’ DNA at a crime scene (fingerprints are so yesteryear.)

Webster’s defines forensics as “The use of science and technology to investigate and establish facts in criminal or civil courts of law.” 

I think this fits well with corporate needs for network forensics using analysis tools.  Many vendors have tools with catchy taglines on variants of “Retrospective Analysis”, “Business Forensics”, “Turn Back the Clock,” and so on. Unlike real-time, the basic premise behind network forensics is to mine data (usually via packets) and perform post analysis to reconstruct content or gather intelligence as to why certain things happened.  In some ways, forensics is like detailed hindsight.

There are several areas where forensics can be applied.  Samples of some broad categories include:

• Compliance:  Oops, someone sent out company confidential financial information in an unencrypted email or used IM to gossip about an coworker's medical condition. 
• Troubleshooting: Why did our network meltdown this morning?  Why do our CRM users often experience poor performance in the afternoon? 
• Verticals:  Why did the core switch peg during a critical trading hour?  Why are doctors losing wireless connectivity?  Is our converged VoIP operating smoothly?

Returning to the Webster definition, analysis tools can be used to establish facts as to a particular network related event that is disturbing.  By network, I mean the entire infrastructure from fabric to nodes to users – let’s not forget about the human element.

According to searchnetworking.com, Marcus Ranum is credited with saying “Network forensics is the capture, recording, and analysis of network events in order to discover the source of security attacks or other problem incidents.”

Thus by virtually all definitions of the term, forensics is traditionally associated with crime solving.  This puts us more in the overall thinking of “when illicit things happened, what happened, and how can it be prevented?” whether from inside or outside sources.  Contrast this to troubleshooting as mentioned above.  Forensics is criminal.  A slow network is not.  Although one could argue that a dead network is criminal!

There’s also a relatively new category of forensics – enterprise forensics that focuses both on user activity and what drives or doesn’t drive the business (analytics + behavior).  Is what we’re seeing on the network consistent with business objectives?  Apdex (a measure of application satisfaction with respect to end-users) is a good measure of this.  Now we’re crossing the boundary over into Application Performance Management (APM) along with user behavior and IT vs. business expectations.

Some companies in fact have attempted with limited success, to focus on the behavioral aspect of forensics.  According to the book Digital Evidence and Computer Crime, “Behavioral evidence analysis provides a systematized method of synthesizing the specific technical knowledge and general scientific methods to gain a better understanding of criminal behavior and motivation.”

The big question here is whether or not unusual behavior can be predicted in advance based on characteristics of anomalous activity.  So what if Johnny does a 2 gig file transfer on Friday afternoon?  Maybe it’s a routine back-up.  Maybe it’s a one-off OS patch.  Furthermore, who cares if our WAN utilization is 100% during that time, as long as it’s fair access for all and all access for one when no one else is on at the moment?  Naturally, illegal file sharing and such is cause for alarm, but there will always be unpredictable anomalous spikes that don’t fit baselines.

I used to teach in my network analysis and troubleshooting courses that having 100% utilization is not necessarily a bad thing and should not be sending and setting off SNMP traps and alarms all over the place.  Of course, you don’t want one user or application to consume all the bandwidth for extended periods of time.  But brief bursts of 100% can actually be a good thing.  After all, if no application can ever utilize 100% of the pipe, then perhaps we should optimize things.  The key is to get on and off the network as quickly as possible – big pipes (and low end-to-end latency) help to achieve that.  But I digress.

Forensics tools need to provide the flexibility to blend real-time analysis with forensics and allow you to optimize the tool for your given situation.  Is 100% capture to disk of massive amounts of data important to you?  Where do you need to cover (capture) in your network, what is the nature of the traffic, and what are the capture bandwidth requirements?  How long do you need to keep the data around?  How important are the distributed aspects and how efficient is the data conveyed to centralized consoles or distributed consoles shared by multiple engineers (investigators)?  Do you prefer that the forensics data mining and subsequent analysis be carried on at the remote engines or brought back to the console to analyze locally and/or take off-line?

So slip on a mask and a pair of latex gloves and get to work!

SpyNet Showcase

My good friend over at Gigamon, Denny Miu, called me some time ago to inform me about an exciting new showcase at the upcoming Interop show in Las Vegas (May 22-24th).  As soon as I heard the word “SpyNet” and knowing the business Denny is in, I immediately jumped on board.  Then he filled me in.

By the time Interop Hotstage (where the supporting show network and NOC as well as iLabs networks are assembled) rolled around, the remaining analysis vendors flocked to Spynet like a honeypot.  All the major network analysis competitors will now be racked together at Interop, a first.

This is a good thing.  I welcome competition.  The whole setup and participation conjures up images in my mind of the Spy vs. Spy thing from Mad Magazine.

YOU benefit.  Come and see how data is collected and analyzed.  All vendors will be tapping off Denny’s orange octopus (GigaVue) to analyze the show traffic to and from the Internet either side of the Interop NOC firewall fed by dual 45 Mbps circuits.  Stop by the Spynet showcase right next to the iLabs booth then come by our booth, 1966, and you’ll see what we see live.  It promises to be interesting stuff.  Intrigued?  See you there!

802.11n Going Enterprise?

With the announcement this week from Meru that it will demonstrate new gear based on the 802.11 draft n specification at Interop and ship this summer, is it fair to say that “n” has entered the enterprise?

Before we get too excited, let’s look at where we are today and then come back and take another look at the Meru announcement.  Quite frankly, things are looking promising but there are some caveats.  First, for the good news.

We recently conducted a number of lab tests with new draft n gear and the results were encouraging as far as interoperability.  For the first time, MIMO gear of different manufacturers are actually talking to each other.  Great!

The test included Linksys business class APs and client adapters, as well as D-Link and Netgear, operating in 802.11n only mode.  The hardware included a mix of Marvell and Atheros chipsets.

With all the major laptop vendors including Dell, HP, Toshiba, and Lenovo shipping draft-n in new laptops and cards available from all the 2nd-tier networking vendors, can enterprise 802.11n be far behind?  Many are waiting for the Cisco shoe to drop.  If history repeats itself, Cisco is typically the last to enter a new market. They traditionally wait for standards to finalize before shipping new product.

Meanwhile, to benefit early adopters (both vendors and users), the Wi-Fi alliance (of which WildPackets is a member) will certify draft-n (Draft 2) interoperability later this year.  The Wi-Fi alliance, as you may recall, gives us an early alternative, but complimentary to the 802.11 standards such as WPA before 802.11i (and now Wi-Fi WPA2), WMM before 802.11e, and so on.

Now for the caveats.  While one can benefit from MIMO on a single 20 Mhz channel, 802.11n requires two channels for maximum throughput.  Using OmniSpectrum during our tests, we saw that the 2nd channel allocated was 4 channels from the primary channel – i.e. channel 11 as primary and 7 as secondary, 1 primary, 5 secondary, etc.

This impacts enterprise 802.11n roll-outs.  For instance, the old honey-comb coverage rule of 1-6-11 in the 2.4 Ghz b/g band need no longer apply.  Think about it.  At 300 Mbps using 40 MHz and two streams, there is only one non-interfering, non-overlapping deployment possible!  The best all 802.11n coverage may be a 1 and 5 and 7 and 11 scenario, but channels 5 and 7 are still too close together to be practical.

And of course there are millions of legacy devices to deal with that will be with us for awhile.  How to stagger and optimize coverage for n and b/g?

The ideal solution?  Ressurect the 5 GHz 802.11a band (which is an important aspect of 802.11n draft 2). The 5 Ghz band certainly has some benefits including staying out of the way of b/g and it’s able to accommodate more n channels (up to 3 non-overlapping in fact).  Unfortunately, 5 GHz signals peter out quicker than 2.4 GHz given the same transmit power (i.e. you get less distance with 5 GHz), so we are back to a similar problem we had with b/g vs. “a.”

Meanwhile, there is at least some performance gain even for minimal single channel n (130 Mbps promised with 2 stream MIMO) deployment.  In fact Intel is pondering only single channel support to start with Centrino n, because of all the b/g legacy concerns.  Some 802.11n APs will also be smart enough not to offer the 2nd channel when b/g is present.  But again, this speed bump effectively throttles n when in use in the 2.4 Ghz band.

Back to Meru.  Their new AP300 can carry up to two 802.11n radios or one 802.11n and one a/b/g radio.  Interestingly, Meru will support 802.11n in the 5 GHz band, but where are the clients?  All of the aforementioned laptop vendors currently support draft n in the 2.4 GHz band.

For of the majority of Enterprises that have rolled out b/g, it seems the best we can hope in the short term is to support pockets of selected 802.11n users that need the boost.  Enterprises that have included 5 GHz in their plans and have the infrastructure to support it may be in better shape to take on 802.11n for total coverage– provided the clients are there.  I would not be surprised if Cisco announces support for 802.11n in the 5 GHz band before year end.  If so, expect a torrent of new client (and AP) hardware to follow from them and many others.