Network Analysis Unplugged

The Year in Review... and Ahead

Among my predictions for 2007 were continued growth in WiFi with 802.11n settling down, accelerated developments in 10 Gigabit due to passing of the 10GBASE-T copper media standard, accelerated data analysis on large volumes of captured traffic, advancements in forensics, and a couple of miscellaneous other things.

Of the above, accelerated data analysis and advancements in forensics fell far short of my expectations.  The only real advancements were in the sheer volume of storage offered by various data mining probes from the likes of Network General/NetScout, Network Instruments, Niksun, Solera, WildPackets, etc.  Besides SANs, these advancements essentially piggybacked the astonishing advancements in 3.5” hard drive technology, namely higher capacity and lower cost, thanks to Perpendicular Magnetic Recording (PMR).  The hardware is down to twenty cents a gigabyte.  Unreal.

Meanwhile, where were advancements in analyzing terabytes of packets? Better search tools?  Faster results?  Smarter expert systems? In the field, I continue to apply my own techniques over a variety of tools to achieve the results I’m looking for.  The vendors have disappointed in this area.  Enough with the whining and on to predictions for 2008.

More Network Analysis Acquisitions.  Fluke started off the year by announcing its acquisition of Crannog.  More recently, analysis vendors like Network Physics and Network General were snapped up for a song.  What will be the first analysis acquisition of 2008?  Possibly privately held performance analysis vendor NetQoS, but not for a song.  The company is showing some nice growth as late and the rumor on the street is that IBM is seriously courting.  Other analysis companies with acquisition potential include Network Instruments, WildPackets, and one or two others.  Stay tuned for an interesting year.

Virtualization Buzz Levels Off.  Datacenter consolidation, growth in blade servers, the green movement, multi-core processors, cheap high density memory and cheap storage added to the virtualization hyperbole in 2007.  Sure it will be hot in 2008, but highly commoditized making it a household word.  I’m not sure what will be hot and new in virtualization in 2008.  Maybe a push to deploy those analysis engines on the virtual machine (to capture packets from the virtual Ethernets).  Minor problem #1: Where to store all that data?  Minor problem #2: How to analyze all that data without impacting the core CPUs already running all those virtual machines?  One solution is for the analysis engine (or stand-alone for that matter) running in the VM and capturing packets off virtual networks to share a high performance back-end data store.  Let’s hope that 2008 will finally bring us some real innovation in high volume data analysis and forensics.

802.11n Soars, Analysis Lags.  No surprises here.  802.11n Draft 2.0 passed early this year, multi-vendor interoperability is real, the WiFi alliance has certified some 200+ Draft 2.0 products, Cisco announces its push into the enterprise, and 300 million WiFi chipsets were shipped in 2007, up 41% from the previous year.  802.11n will also resurrect interest in deploying WiFi in the 5 GHz band where it makes the most sense to not only stay out of the way of legacy 802.11 b/g and general interference, but to also provide more spectrum for non-overlapping dual channel 40 MHz coverage.  New wireless analysis challenges will result, especially in monitoring highly location dependent reception of dual channel MIMO traffic.  Stationary wireless monitoring tools are inadequate (except in niche situations) and 802.11n portable laptop analyzers can easily lose a lock on promiscuous MIMO reception while moving around, unlike their 802.11 a/b/g brethren.

10Gig (and TAPs) Take Off.  I’m cheating a little since at the end of 2006 I already figured 2008 to be the year of 10 Gigabit Ethernet.  So I’m making it official.  Look for this to be the big year as well as unprecedented announcements from TAP vendors in the areas of density, price drops per port, and new products.  Look for TAPs to be smarter too (filtering, buffering, stream aggregation, stream splitting, timestamp generation, etc), to help offload the strain from those aforementioned overloaded terabyte analysis probes that are dropping packets like flies.  Come to think of it, perhaps this is where the next innovation in analysis and forensics will come from - the TAP vendors.

Quality of Experience (QoE) Takes Center Stage.  The buzz surrounding quality of service (QoS) for not only VoIP but data applications as well and how it affects the end user QoE will continue to garner attention in the coming year.  I’m working on an interesting enterprise network analysis process showing how faster infrastructure response times in a series of tasks leads to a faster response from the user in between the tasks. User behavior needs to come into the equation along with how servers react to certain events, something that cannot be easily measured by TCP transactions alone.  Another area to keep a close watch on is how highly tuned converged networks carrying large volumes of VoIP traffic can impact data.  Have we forgotten about our mission critical data applications on the path to VoIP?  There’s so much more to QoE that I can’t possibly see how it will escape attention in 2008.

I wish you all the best in the New Year!

Got Analysis Tools? Put a Shark in Your Tank

As the year winds down this snowy December day at a balmy 18 degrees (I’m writing this from an undisclosed cabin “up north” in Minnesota), my mind is far from the cardinals, nuthatches, and woodpeckers just outside my window.  In fact, I’m thinking about sharks.

For me, no one analysis tool fits all.  I typically use a wide set of tools: OS command line utilities, open source projects, commercial data mining and analysis products, resource monitoring utilities, and applications like Excel.  Being resourceful and combining tools is the key to quick and effective enterprise infrastructure analysis. 

I prefer the term infrastructure analysis because it avoids saying, “Is it the network or server?” When it comes to troubleshooting a problem, I look at everything starting from the way devices talk to each other (device and even user behavioral analysis based on protocol analysis) and digging in or drilling down from there.  Sometimes I get lucky and find a simple physical connection duplex mismatch causing CRC errors leading to TCP retransmissions. Other times it gets far more complex, like when I get into n-tier analysis with a multitude of devices, protocols, server types, and applications. But I digress.

Regarding the shark, as you probably guessed I’m referring to Wireshark (formerly Ethereal), updated this month to version 99.7.  I would not recommend depending solely on Wireshark unless you have time to burn doing stuff manually that could be done much faster using other tools. For that matter, some data you just can’t get with any protocol analyzer.

But the shark is in my tank for good reason.  I suppose one reason is that it’s free.  But time is money and free tools can actually be more expensive compared to shelling out a few (thousands) dollars for commercial offerings if the commercial products save you valuable time. But the shark is quite good for certain tasks.

One of the more obvious reasons to use Wireshark is for its breadth and depth of decodes.  Believe it or not, some analyzers actually have deeper decodes for a few protocols, but none have the breadth.  For example, in a recent analysis of a large enterprise, I needed decodes for Distributed Relational Database Architecture (DRDA), a standard driven by the Open Group and IBM for accessing distributed data based on SQL.  While the SQL statements could be read as plain text in the TCP payload, it helped to know in what context these commands were being used.  A couple of other unnamed commercial analyzers that I had access to did not decode it.  The shark to the rescue.

A not-so-obvious use for Wireshark is packet conversion—read in one format and write to another.  Observer native format to OmniPeek native format?  No problem.  Most analyzers support the simple Sniffer .enc format as well as their proprietary format.  The problem is that some do not do a good job of converting to .enc, resulting in packets that are flagged as “sliced” but really aren’t or have the occasional timestamp conversion problem.  Wireshark seems to do a pretty decent conversion to .enc, at least a lot better than one unnamed commercial offering.

The shark is also portable.  I’m not so much referring to its multi-OS portability as I am to its USB flash drive portability.  I can run a version of Wireshark right off a U3-compatible USB flash drive without having to install it on the host system.  A shark in your pocket.  How cool is that?

There are numerous other tricks for the shark that can complement your analysis including some cool command line stuff.  You can learn from the experts that live and breathe the shark at the forthcoming Sharkfest that promises to be more exciting than, well, the original Jaws classic.  The event is March 31st – April 2nd in Los Altos Hills, CA (near Sunnyvale). 

I can envision the feeding frenzy if the highly energetic Laura Chappell shows up dressed in a shark outfit with a big TCP FIN on her back.  Ouch.

Inside 802.11n Frame Aggregation

I’ve been busy lately working on a white paper entitled "Inside 802.11n Wireless LANs: Practical Insights and Analysis” now available for download. In the section ”Cutting the Overhead” I detail how 802.11n has improved the efficiency of the MAC protocol using a single block ACK (BA) for multiple frames as well as the ability to aggregate multiple frames into a single transmission.  I’ve excerpted information on frame aggregation in this blog.  For insight into the block ACK along with frame capture analysis, please consult the white paper.

802.11n can send multiple frames per single access to the medium by combining frames together into one larger frame.  There are two forms of frame aggregation: Aggregated Mac Service Data Unit (A-MSDU) and Aggregated Mac Protocol Data Unit (A-MPDU).

A-MSDU increases the maximum frame transmission size from 2,304 bytes to almost 8k bytes (7935 to be exact) while A-MPDU allows up to 64k bytes.

A-MSDU creates the larger frame by combining smaller frames with the same physical source and destination end points and traffic class (i.e. QoS) into one large frame with a common MAC header.  One way to visualize this is an access point receiving frames from the wired side at a rate faster than it can transmit them on the wireless side.  Ethernet frames headed for the same wireless client can be queued then combined into one larger frame for single transmission, cutting down the overhead dramatically.

The Caveat

One caveat, and it’s a big one, is that like any wireless transmission, the larger the frame the less likely it will be received with no errors.  I’ve observed that 802.11n senders tend to learn the maximum data rates possible for given frame sizes.  Thus for instance, you may see a frame containing a TCP ACK send at 270 Mbps (or 300 Mbps if a short guard interval is in use) because it’s very small (typical 78 bytes including the 802.11 overhead).  Frames containing FTP data may be sent at 121.5 Mbps simply because previous attempts at a higher data rate proved futile.

Thus it’s not clear how much we will gain in efficiency using the A-MSDU.  With its frame size up to 7935 bytes, with one MAC header and payload protected by one CRC like any other single frame, it will most likely be transmitted at a lower data rate for reliable reception.

Contrast this to the A-MPDU which is essentially a chain of individual 802.11 frames sent back-to-back with one access to the medium (i.e. one preamble).  The destination must still be one address and the traffic class (QoS) must be the same for each.  Clearly, there is more overhead with the A-MPDU because we still have individual PDU frame headers vs. one in the A-MSDU.  Unlike the A-MSDU however, individual PDU frames also have their own CRC; an error in one PDU will not affect the others in the group.

The bottom line is this:  Reliability has its overhead.  The A-MPDU allows a much larger "burst" of frame data to be sent compared the the A-MSDU.  In fact, many 802.11n access points conservatively option for the 1/2 A-MSDU frame size (3839 bytes) by default to play it safe.