Network Analysis Unplugged

Analysis in the Enterprise - A New Series for the New Year

Admittedly, 2008 has been a quiet year for me on the blog front.  I've had an extremely busy year on the IT front, however, spending virtually all of my professional time at U.S. Bank working on large scale application problem and performance projects.  It has been a challenging year in a highly varied and complex network and application environment where I've had a lot of fun learning new analysis techniques (who said you can't teach an old dog new tricks?) and contributing in new ways. 

And when it comes right down to it, my favorite tools continue to be those that break down packets, allowing me to study in detail the application and service behavior once I get past the mundane but important fundamentals ranging from physical layers problems to TCP retransmissions to HTTP response time.  We used to call these tools that empowered us analysts "protocol analyzers."  Now they have fancy names like "OmniPeek", "InfiniSteam", "Performance Manager", "Gigastor", and even "WireShark."

In the weeks and months to come, I plan to write more frequently and on topics you wont'f find anywhere else.  I'll continue to offer tips and tricks but focus on industrial strength Enterprise analysis, moving off the typical client/server analysis.  A sampling of topics include troubleshooting applications that utilize enterprise protocols like Message Queuing (MQ), ConnectDirect, and Distirbuted Relational Database Architecture (DRDA), challenges with 10 Gigabit analysis aside from the obvious (speed and volume), new insights into SPANs vs. TAPs, how to use packet analysis tools to study user behavior, and why even the top vendors still don't get the enterprise. 

Yes, I know that that last one sounds a bit harsh.  I can empathize as I was the CTO at one of those vendors for seven years.  Now I can get on my soap and express an opinion or two (but not just on vendor related topics).  ;-)

So fans, I'm back for 2009.  I'm making it a goal to continue to make this blog informative and fun, and share knowledge based on real world experience.  I hope you enjoy it.

Live Webinar and Survey Reveals Wireless Secrets

I recently attended a live Cisco Mobility TV webinar co-sponsored by AirMagnet entitled "Designing and Deploying 802.11n Next-Generation Wireless." Apparently it was a big hit; according to the moderator, a record "thousands" of viewers logged in to watch it. Here’s what I thought were a couple of interesting takeaways.

Drop in Replacement or New Site Survey Required?

A Cisco representative started out by recommending a 1-for-1 access point replacement of legacy APs giving priority to performance over coverage. In other words, swap in a Cisco dual band 1250 AP to handle both legacy 802.11bg devices with the same coverage pattern as before while providing 802.11n access (in the 5 GHz band) for new 802.11n clients.

I thought this was a bit strange since 802.11bg does not like reflections whereas 802.11n using MIMO thrives on it. APs must be relocated accordingly. Later in the Webinar, the AirMagnet guy noted that an active site survey using a laptop with a live 802.11n client adapter is required to figure out the optimal 802.11n AP placement to take advantage of multipath. This seemed to contradict the Cisco 1-for-1 forklift strategy.

AirMagnet also recommended surveying with more than one 802.11n client adapter type if you are supporting more than one brand. I think this is a good idea since, unlike 802.11bg, Cisco does not provide a client side adapter.

Upping the Power Requirements

Perhaps the most controversial aspect of an 802.11n wireless LAN upgrade is the additional power requirements for dual band APs. Cisco claims that their enhanced PoE is the only viable single port solution for dual radio operation. They stated that dual PoE is twice the cable cost, 4x the cost to pull the cable, and uses more switch ports. They also noted that competitors supporting dual band operation over standard 802.3af PoE do so at reduced transmission power.

Luckily, CPUs are not the only chips going green. Witness the recent announcement from Siemens that generated a flurry of online articles discussing the 802.11n power controversy. Siemens managed to cut 3W off a full 802.11n MIMO (roughly 600 Mbps using transmission over 3 antenna with 3 streams each) AP running at maximum radio output in both the 2.4 GHz and 5 GHz bands simultaneously and yet operate over standard PoE.

Audience Survey

Having a captive audience of thousands, Cisco conducted three polls during the Webinar. Here are the questions and results.

"How do you expect to benefit from the use of 5 GHz with 11n?"

No surprises here. Only 5% claim that they will not use 5 GHz for 11n, vindicating the use of 5 GHz in the enterprise.

"What do you see as the biggest inhibitor to 11n adoption?"

Is the undetermined business need from those not deploying wireless whatsoever or are their needs currently met by 802.11bg? Also note the lack of a warm fuzzy for the current 802.11 draft 2.0 standard.

"How do you plan to power your 802.11n access points?"

Not much to add here. Looks like the largely Cisco audience prefers enhanced PoE.

So there you go. Some inside info and survey results from a wildly popular wireless webinar.

The VoIP MOS Debacle

I recently posted a feature article on LoveMyTool that deals with the inconsistency in Mean Opinion Scores (MOS) as reported by packet-based VoIP analysis tools.  MOS is a rating system for voice transmission quality as determined by a group of human listeners. Computer generated versions attempt to emulate how such a group would rate the call quality.

Speaking from experience, some analyzers are far more accurate than others.  The bottom line is that no one vendor uses the exact same algorithm to compute MOS unless they OEM the code from the same source.  When selecting a tool, insist on test data to back up vendor claims; typically real call data under different scenarios validated against human listeners conducted by a legitimate third party tester.

I highly encourage you to read the full article.

Inside 802.11n Frame Aggregation

I’ve been busy lately working on a white paper entitled "Inside 802.11n Wireless LANs: Practical Insights and Analysis” now available for download. In the section ”Cutting the Overhead” I detail how 802.11n has improved the efficiency of the MAC protocol using a single block ACK (BA) for multiple frames as well as the ability to aggregate multiple frames into a single transmission.  I’ve excerpted information on frame aggregation in this blog.  For insight into the block ACK along with frame capture analysis, please consult the white paper.

802.11n can send multiple frames per single access to the medium by combining frames together into one larger frame.  There are two forms of frame aggregation: Aggregated Mac Service Data Unit (A-MSDU) and Aggregated Mac Protocol Data Unit (A-MPDU).

A-MSDU increases the maximum frame transmission size from 2,304 bytes to almost 8k bytes (7935 to be exact) while A-MPDU allows up to 64k bytes.

A-MSDU creates the larger frame by combining smaller frames with the same physical source and destination end points and traffic class (i.e. QoS) into one large frame with a common MAC header.  One way to visualize this is an access point receiving frames from the wired side at a rate faster than it can transmit them on the wireless side.  Ethernet frames headed for the same wireless client can be queued then combined into one larger frame for single transmission, cutting down the overhead dramatically.

The Caveat

One caveat, and it’s a big one, is that like any wireless transmission, the larger the frame the less likely it will be received with no errors.  I’ve observed that 802.11n senders tend to learn the maximum data rates possible for given frame sizes.  Thus for instance, you may see a frame containing a TCP ACK send at 270 Mbps (or 300 Mbps if a short guard interval is in use) because it’s very small (typical 78 bytes including the 802.11 overhead).  Frames containing FTP data may be sent at 121.5 Mbps simply because previous attempts at a higher data rate proved futile.

Thus it’s not clear how much we will gain in efficiency using the A-MSDU.  With its frame size up to 7935 bytes, with one MAC header and payload protected by one CRC like any other single frame, it will most likely be transmitted at a lower data rate for reliable reception.

Contrast this to the A-MPDU which is essentially a chain of individual 802.11 frames sent back-to-back with one access to the medium (i.e. one preamble).  The destination must still be one address and the traffic class (QoS) must be the same for each.  Clearly, there is more overhead with the A-MPDU because we still have individual PDU frame headers vs. one in the A-MSDU.  Unlike the A-MSDU however, individual PDU frames also have their own CRC; an error in one PDU will not affect the others in the group.

The bottom line is this:  Reliability has its overhead.  The A-MPDU allows a much larger "burst" of frame data to be sent compared the the A-MSDU.  In fact, many 802.11n access points conservatively option for the 1/2 A-MSDU frame size (3839 bytes) by default to play it safe.

The Route Less Traveled: Musings on Layer 3 Switching

According to Wikipedia, the only difference between layer 3 switching and routing is in the internal hardware implementation.  Layer 3 switching takes place in specialized hardware vs. general purpose microprocessors in traditional routers. Furthermore “Layer 3 switches can be placed anywhere in the network because they handle high-performance LAN traffic and can cost-effectively replace routers.”  Thus, the motivation behind layer 3 switching appears be a need for high performance routing in our LAN environments.

Traditionally we built our LAN (and WAN for that matter) infrastructure by dividing our IP address space into subnets, which required inter-subnet routing.  The escalation in LAN bandwidth per user and changing application performance requirements placed additional demand on routers.

But let’s not overlook one very important aspect that had a far greater impact:  Over the years we virtualized our LANs into VLANs.  The original motivation for doing this was to control broadcast domains as our layer 2 switching infrastructure grew.

Dividing our LANs up into VLANs led to new benefits in the areas of security, management, and user control to name a few.  We even dedicated VLANs to applications requiring different handling characteristics such as VoIP.  Some esoteric uses for VLANs included assigning multiple SSIDs to a wireless access point in order to map to VLANs on the wired side.  This allowed us to prioritize incoming wireless traffic, restricting uses to certain resources, or both (corporate vs. guests or faculty vs. students, for instance).  It seems that our imagination became the limit.

Unfortunately we were faced with a rather nasty problem - how to get traffic between these isolated broadcast domains?  After all, packets like ARP broadcasts were stopped cold in their tracks. The answer?  Associate a VLAN with a particular subnet and use a router behind a switch to handle VLAN-to-VLAN traffic.

How efficient!  Not.  Extra packet hops and traffic across LAN segments and significant additional load on the router ensued. This problem became affectionately known as "local routing" and all the expert systems in protocol analyzers detected it (well the good ones anyhow).  Two devices could be plugged in side-by-side in the same layer 2 switch but on different VLANs and subnets.  Instead of containing the traffic inside the switch with high-speed switching between ports, the packet was sent up to a router and back down again, wasting bandwidth and router resources.

So what came next?  We threw more backbone bandwidth and router CPU at it!  I'm only half serious here. But for a while it was the only way to solve the problem and some are still doing it this way today.  Layer 3 switching to the rescue.

By pushing layer 3 switching down to the device per port level and applying techniques similar to content addressable memory found on layer 2 switches, but for IP addresses instead of MAC addresses, we solved our little VLAN problem.  Not only that, but we can now have as many ports as we want dedicated to one VLAN/subnet to serve multiple devices.  No more restrictive one-to-one subnet-to-router port relationships.  Slick.

One thing I really like about layer 3 switches when it comes to network analysis and troubleshooting is that they decrement the TTL in the IP header, just like a conventional router.  Thus allows you can see how many store and forward devices a packet has traversed, even if they were virtual hops within a single layer 3 switch.  Try that in conventional, transparently-switched layer 2 environments.  You have to resort to manually figuring out the switch route or using specialized analysis tools to follow the switch MAC address tables via SNMP. Fluke Networks specializes in this sort of thing, as a matter of fact.

As for layer 3 switching vs. routing in your environment, much of it boils down to how many localized VLANs you have and how much traffic passes between them.  This becomes an interesting exercise in analysis because some applications like VoIP will have a dedicated VLAN up to a certain point (think about scalability) and yet other applications and servers will have their own unique switching and routing requirements.
 
Another interesting question is at what point do you push layer 3 switching down to the workgroup?  I’ll leave that as an exercise for the student.

802.11n Interoperability Jumps to 28 Vendors; Patent FUD?

As anticipated, proponents of 802.11n have something to cheer about.  The Wi-Fi Alliance announced that 95 products have been certified for 802.11n Draft 2.0.  The product mix is from some 28 vendors which is pretty impressive considering the testing program began just three months ago.

Big name chip vendors including Atheros, Intel, and Marvell are in the fray, as well as heavyweights Apple, Cisco, HP, Sony, and Toshiba to name a few.

The Register, a flamboyant UK tech web rag, is the flash point of the latest rumors reported that “…a significant threat to the standard from patents held by the Commonwealth Scientific and Industrial Research Organisation (CSIRO). Despite requests from the IEEE, CSIRO has failed to promise not to sue anyone for infringement.” 

As with any IEEE standard, if any technology incorporated into the standard has any patents, they must be waived by the respective patent holders.  Naturally, the various engineers and companies that contribute to the IEEE try to avoid such conflicts in the first place or get a “Letter of Assurance” not to sue from any patent holders.

Not specified by the “exclusive” Register news what the patent actually covers.  A cursory search of the CISRO web site down under did not reveal any ongoing patent issues with the IEEE.  Wi-Fi Planet reported back in 2005 that the patent in question relates to OFDM.

CISRO is certainly not alone in patents pertaining to technologies employed by 802.11n.  In fact, the aforementioned article states that “The IEEE lists 23 companies with patents related to 802.11n. According to the U.S. Patent Office, there are 634 U.S. patent applications and 255 patents granted regarding MIMO.”  More specifically, the article notes that “Of particular interest is a Speedus Corp. patent similar to one definition of MIMO. U.S. Patent 5,949,793 includes a method "relating to simultaneous transmission of digital and analog signals in the same band."

Have all patent concerns been addressed by 802.11n? Is CISRO unique as the last hold out for a Letter of Assurance to the IEEE?  I doubt it.  I do believe however, that all parties will come to terms – eventually.

802.11n to Impact Ethernet?

In my previous blog, I looked at some of the controversy surrounding a report entitled “802.11n: The End of Ethernet?”   There is no doubt that wireless is already having an impact on Ethernet.  Is 802.11n any more special?

According to the press release for the report, 802.11n is “an appropriate LAN access substitute for wired Ethernet” when:

• The number of laptop users is growing
• The enterprise uses mobile applications
• Fast Ethernet (100 Mbps) throughput is good enough
• The enterprise deploys Voice over Internet Protocol (VoIP)
• Moves/adds/changes are frequently made
• The risk of deliberate denial of service attack is low to moderate
• Ethernet cable installation is difficult

After looking at this list, I’m thinking to myself: Which of these are existing enterprise 802.11 wireless installs failing to satisfy? It seems to boil down to the promise of a faster channel rate offered by 802.11n enabling support for a larger number of mobile and VoIP users.

Sure, Fast Ethernet is adequate for the majority of workstation and laptop users.  The problem is that we’ve migrated to a dedicated 100 Mbps switch port for every user.  With single channel 802.11n MIMO operation we’re looking at a raw bandwidth of 150 Mbps.  In single user lab tests, a file transfer can crack 100 Mbps.  Single user lab tests are not real-world.

The real problem is that if we move away from our established one-port per user switched model, we’re going back to the days of shared Ethernet, only worse. With 802.11n,  not only is bandwidth contention a factor, but other issues affect performance as well:

• Slower data rates at distances farther away from the access point
• Environmental interference (not necessarily other RF sources, but physical obstructions and attenuators)
• Having to manage QoS policies to support VoIP over wireless
• Being subject to RF jamming, and so on.

In other words, many of the same issues we’ve faced with any WiFi technology to date.

Going to 300 Mbps dual channel operation will help, but has many additional issues including less spectrum to deploy multiple non-overlapping access points (a requirement to afford more coverage area as well as drop the number of users sharing channels).   Utilizing the 5 GHz “a” band will certainly help with the spectrum requirements but, unfortunately, reduces distances.  And how about laptop support for 5 GHz dual channel’s additional  power requirements, the more complex antenna systems, complete interoperability at the high end, etc.?  The jury is still out.

There is no question that wireless will continue to impact Ethernet by way of eroding the Ethernet component market which appears to be the crux of the report.  The title was merely a grabber.  The real answer is much more complex.  For instance, how do laptops really come into play?  What happened to the prediction from a couple of years ago that laptops would replace desktops?  But I digress.

Are we ready to go back to shared Ethernet?

802.11n: The End of Ethernet? Not!

Controversial reports are designed to generate lots of media attention.   A report from Paul DeBeasi at the Burton Group is no exception.  There’s been continued buzz surrounding a report entitled “802.11n: The End of Ethernet?”  With all the blogs I’ve been writing about 802.11n, I wish I had thought of it.

In my case, I would have used the attention grabbing title to state why it would not be the case.  I figured with all the publicity surrounding the report, why not provide some counterpoints?

The report claims that 802.11n marks the beginning a rapid shift away from LAN deployments.  It does not state when this beginning is or was.  802.11n is still in draft stage.  If anything, the recent Cisco 802.11n product announcement just this month marks the beginning, but the report was released back in June.  So where’s the mark?

DeBeasi is quoted as saying “802.11n will put pervasive mobility on the fast track.”  Wait a second.  I thought WiFi period but pervasive mobility on the fast track. Remember when laptops required PCMCIA cards to connect to Ethernet and now every laptop has Ethernet built-in?   The same thing has already happened to with wireless.  Virtually every laptop built in the past year or so has built-in 802.11 b/g.  Thus, they are already enjoying pervasive mobility today.

Don’t get me wrong.  I’m bullish on 802.11n.  So what is it about 802.11n that spells doom for Ethernet, according to the report?

Tomorrow, I’ll provide you with some additional insight and counterpoints.

Cisco Does 802.11n – For Real

Cisco has dropped the 802.11n shoe by announcing their first enterprise level wireless gear meeting the 802.11n Draft 2.0 specification (their consumer division, Linksys, has been shipping draft n gear for some time now).

The WiFi Draft n certified access point has been dubbed the Aironet 1250 Series.  This AP can run in unified (LWAP) or autonomous (stand alone) mode. It looks like unified operation requires the new Cisco Wireless Service Module (WiSM) which pops into a Catalyst 6500 as well as Cisco Unified Wireless Network 4.2 software.

Wisely, Cisco is not immediately offering their own client card.  Let’s face it – there are simply too many notebook options out there (PC Card, ExpressCard, USB, built-in, and so on.)  Instead, they have been working with Intel to pre-test, certify, and offer new Client 5.0 software that will work with the Centrino 4965agn chipset.

Another point of interest is that running the power hungry 1250 AP on Power over Ethernet (PoE) will require a blade upgrade in existing 3750, 4500 and 6500 Catalyst switches.

My friend Frank Bulk over at Network Computing Online, says this about performance: "Sites <with Cisco> have seen rates of 120, 130, and even has high as 138 Mbps per radio. While this is likely in a greenfield environment without the debilitating effects of legacy clients, these claims did exceed by a large margin those privately demonstrated by both Meru and Trapeze at Interop Las Vegas."

Back in May I pointed out the optimism with 802.11n Draft 2.0 – the first version with real hope for interoperability - as well as various caveats and why it will resurrect interest in the almost forgotten 802.11a 5 GHz band.   In fact, Intel has previously stated that the aforementioned Centrino chip will only support 40 MHz (two 20 MHz channels) operation in the 5 GHz band.

It certainly seems though Intel provided the catalyst this time around.

Cisco 250 AP, 802.11n whitepaper, more
All about the Intel 4965agn
Network Analysis Unplugged:  802.11n Going Enterprise?

802.11 RF Survey Patents Heat Up

Motorola has acquired some interesting RF management software – and apparently some patents – with its acquisition of Wireless Valley back in December of 2005.  Add the acquisition of Symbol Technology in January of this year and you’ve got some serious clout in the burgeoning 802.11 market.

So what’s next?  Protect that IP.  Moto has launched a lawsuit against Aruba to stop them from using certain technologies pertaining to 3D site surveying and predictive design along with a pair of related patents from Symbol dealing with wireless LAN switching. To add insult to injury, Moto is adding damages for past use.  It seems that that some prior licensing attempt must have fallen through.

Aruba turned an IPO earlier this year and is apparently doing quite well financially as of the last quarter.  Aruba’s stock had jumped 48% since the IPO, but after Moto's Press Release (apparently after the market closed last Friday) Aruba's stock fell when the market reopened following Labor Day.  There was some some nervousness later in the day with a weak attempt at a rebound (I'm starting to talk like a stock analyst).  The net affect on the first trading day after the announcement was a 6% drop, so I guess the news wasn't TOO bad as yet.

The lawsuit will be interesting to keep an eye on as there are numerous other vendors in the RF site survey space including AirMagnet, AirTight, Cisco, and Ekahau.  Motorola holds a ton of patents.  One can’t help but wonder how much licensing adds to it's bottom line.

The Original Press Release
Motorola’s Wireless Valley Technology
Some nice 802.11 site survey white papers from Motorola (with no registration to boot)