Network Analysis Unplugged

Optimize Your Web Site with YSlow

Some of the best free tools in life are the niche ones such as this nifty little utility available from the Yahoo! Developer Network called YSlow (a nice play on Why Slow). You'll need to install Firefox along with the Firebug web development tool to run it, but it's definitely worth it. I'll first describe a controversial aspect of the tool, then get on to the interesting stuff.

The Controversial Part

YSLow will analyze a web page and assign a grade of A through F based on 14 criteria. For a complete list of the criteria, check out "Best Practices for Speeding up Your Web Site." Some of the grading criteria is straight forward like the number of HTTP requests or turns (the fewer the better). Others are more complex and designed for larger enterprises such as having a Content Delivery Network (CDN) like the Akamai provides for the likes of Apple, Dell, and IBM. Thus if you have a small to medium sized business, you'll get dinged for not having a CDN.

You'll also get nicked for not using something known as "far future Expires header", GZIP components (even for small objects), and ETags. ETags allow web servers and browsers to figure out if an object in the user's cache matches the one on the server. This is somewhat controversial as even the YSlow developers admit that they are "unique to a specific server hosting a site." Please check out the aforementioned "Best Practices" for more details.

Naturally, the Yahoo! Web site gets an "A" grade which is interesting, considering there's far more junk on the page then a lightweight page like Google. Yahoo! used to be lightweight, but thanks to Expires headers, cache priming (i.e. the biggest hit is when you visit the site for the first time), they manage a good score and load very quickly on subsequent visits. One area that could be improved as YSlow points out is to reduce the number of HTTP requests.

Just for fun, I decided to see what the YSlow score was from the very vendors that sell stuff to help us to troubleshoot our network and application performance. It surprised me that the web home pages from companies like Fluke Networks, OPNET, Network General, Network Instruments, NetQoS, NetScout, Niksun, and WildPackets all received a grade of D or F.

Embarrassing? Perhaps. Even without a CDN, one can get much better than a failing grade by employing some of the other techniques. Keeping your pages fairly lightweight and minimizing the number of HTTP requests can go a long ways towards improving the score. Read on.

The Good Stuff

YSlow will show the total bytes and total time to download a page. For instance, the OPNET website weighed in at a whopping 501k bytes and most surprising, 4.0 seconds to download even over a fast 8 Mbps connection. These timings were with a primed cache (i.e. I merely did a page refresh after already visiting the side). Why so long? YSlow to the rescue.

YSlow will show you the timing for each object downloaded, for each TCP session that the browser supports (both IE and Firefox support two TCP sessions to a Web server by default.) The following screenshot of the YSLow "Net" tab gives you an idea of what this looks like. (In the interest of size and readability, I'm only showing you a portion of the web page and timing analysis.)

YSlow Component Load Time Analysis

You can expand the information for each component, including the source code from where it came and even see what the object (such as a gif image) looks like in isolation in another browser window. Each component (HTML, CSS, images, Flash, etc.) can be inspected in detail.

A closer look at the timing analysis revealed that the web page has a large number of HTTP requests including nine external JavaScript files. In all, there were 40 HTTP requests generating 508k bytes worth of traffic that make up this web page, yet roughly 501k bytes are downloaded every time even with a primed cache. Why?

YSlow revealed that only 7k bytes worth of objects are loaded from the user's browser cache. The biggest culprit was the flash animation, weighing in at 477k bytes and downloaded each and every time. This, and the large number of HTTP turns (37 according to a packet capture) coupled with round trip network delay, leads to the longish four second load time for the page even for just a refresh.

How about after clearing the browser cache? Does it get worse? Surprise! There was little real difference. I saw one more turn count and a bit more data. The HTTP turn count is there regardless to check the browser cache against the server and/or download the objects, most of which were tiny gif images or small JavaScript files so it didn't make a whole lot of difference in this case.

Furthermore, the round trip delay (56 ms in this example) starts to become a factor. Fewer turns would help. It also wouldn't hurt to optimize the flash. I'll leave the rest of the web page optimization as an exercise for the student. As a hint, not all web pages that use animation are penalized. The WildPackets home page for instance, went from 471k to 47k when cached (the next figure shows how YSlow gives you these stats). The animation stays cached. The number of objects and HTTP subsequent requests however, are quite high at 62. That's a lot of turns and should be optimized.

YSlow Cache Analysis: Note the drop in bandwidth, but high turn count.

Conclusion

Let's not get complacent just because we can throw more bandwidth at it; i.e. the Internet pipes are getting fatter all the time. Web sites can and should be optimized. If everyone did so we could save, oh, perhaps a few billion bytes worldwide every minute or so. Even corporate intranets need to consider web tuning to best serve up their users.

The VoIP MOS Debacle

I recently posted a feature article on LoveMyTool that deals with the inconsistency in Mean Opinion Scores (MOS) as reported by packet-based VoIP analysis tools.  MOS is a rating system for voice transmission quality as determined by a group of human listeners. Computer generated versions attempt to emulate how such a group would rate the call quality.

Speaking from experience, some analyzers are far more accurate than others.  The bottom line is that no one vendor uses the exact same algorithm to compute MOS unless they OEM the code from the same source.  When selecting a tool, insist on test data to back up vendor claims; typically real call data under different scenarios validated against human listeners conducted by a legitimate third party tester.

I highly encourage you to read the full article.

The Year in Review... and Ahead

Among my predictions for 2007 were continued growth in WiFi with 802.11n settling down, accelerated developments in 10 Gigabit due to passing of the 10GBASE-T copper media standard, accelerated data analysis on large volumes of captured traffic, advancements in forensics, and a couple of miscellaneous other things.

Of the above, accelerated data analysis and advancements in forensics fell far short of my expectations.  The only real advancements were in the sheer volume of storage offered by various data mining probes from the likes of Network General/NetScout, Network Instruments, Niksun, Solera, WildPackets, etc.  Besides SANs, these advancements essentially piggybacked the astonishing advancements in 3.5” hard drive technology, namely higher capacity and lower cost, thanks to Perpendicular Magnetic Recording (PMR).  The hardware is down to twenty cents a gigabyte.  Unreal.

Meanwhile, where were advancements in analyzing terabytes of packets? Better search tools?  Faster results?  Smarter expert systems? In the field, I continue to apply my own techniques over a variety of tools to achieve the results I’m looking for.  The vendors have disappointed in this area.  Enough with the whining and on to predictions for 2008.

More Network Analysis Acquisitions.  Fluke started off the year by announcing its acquisition of Crannog.  More recently, analysis vendors like Network Physics and Network General were snapped up for a song.  What will be the first analysis acquisition of 2008?  Possibly privately held performance analysis vendor NetQoS, but not for a song.  The company is showing some nice growth as late and the rumor on the street is that IBM is seriously courting.  Other analysis companies with acquisition potential include Network Instruments, WildPackets, and one or two others.  Stay tuned for an interesting year.

Virtualization Buzz Levels Off.  Datacenter consolidation, growth in blade servers, the green movement, multi-core processors, cheap high density memory and cheap storage added to the virtualization hyperbole in 2007.  Sure it will be hot in 2008, but highly commoditized making it a household word.  I’m not sure what will be hot and new in virtualization in 2008.  Maybe a push to deploy those analysis engines on the virtual machine (to capture packets from the virtual Ethernets).  Minor problem #1: Where to store all that data?  Minor problem #2: How to analyze all that data without impacting the core CPUs already running all those virtual machines?  One solution is for the analysis engine (or stand-alone for that matter) running in the VM and capturing packets off virtual networks to share a high performance back-end data store.  Let’s hope that 2008 will finally bring us some real innovation in high volume data analysis and forensics.

802.11n Soars, Analysis Lags.  No surprises here.  802.11n Draft 2.0 passed early this year, multi-vendor interoperability is real, the WiFi alliance has certified some 200+ Draft 2.0 products, Cisco announces its push into the enterprise, and 300 million WiFi chipsets were shipped in 2007, up 41% from the previous year.  802.11n will also resurrect interest in deploying WiFi in the 5 GHz band where it makes the most sense to not only stay out of the way of legacy 802.11 b/g and general interference, but to also provide more spectrum for non-overlapping dual channel 40 MHz coverage.  New wireless analysis challenges will result, especially in monitoring highly location dependent reception of dual channel MIMO traffic.  Stationary wireless monitoring tools are inadequate (except in niche situations) and 802.11n portable laptop analyzers can easily lose a lock on promiscuous MIMO reception while moving around, unlike their 802.11 a/b/g brethren.

10Gig (and TAPs) Take Off.  I’m cheating a little since at the end of 2006 I already figured 2008 to be the year of 10 Gigabit Ethernet.  So I’m making it official.  Look for this to be the big year as well as unprecedented announcements from TAP vendors in the areas of density, price drops per port, and new products.  Look for TAPs to be smarter too (filtering, buffering, stream aggregation, stream splitting, timestamp generation, etc), to help offload the strain from those aforementioned overloaded terabyte analysis probes that are dropping packets like flies.  Come to think of it, perhaps this is where the next innovation in analysis and forensics will come from - the TAP vendors.

Quality of Experience (QoE) Takes Center Stage.  The buzz surrounding quality of service (QoS) for not only VoIP but data applications as well and how it affects the end user QoE will continue to garner attention in the coming year.  I’m working on an interesting enterprise network analysis process showing how faster infrastructure response times in a series of tasks leads to a faster response from the user in between the tasks. User behavior needs to come into the equation along with how servers react to certain events, something that cannot be easily measured by TCP transactions alone.  Another area to keep a close watch on is how highly tuned converged networks carrying large volumes of VoIP traffic can impact data.  Have we forgotten about our mission critical data applications on the path to VoIP?  There’s so much more to QoE that I can’t possibly see how it will escape attention in 2008.

I wish you all the best in the New Year!

Got Analysis Tools? Put a Shark in Your Tank

As the year winds down this snowy December day at a balmy 18 degrees (I’m writing this from an undisclosed cabin “up north” in Minnesota), my mind is far from the cardinals, nuthatches, and woodpeckers just outside my window.  In fact, I’m thinking about sharks.

For me, no one analysis tool fits all.  I typically use a wide set of tools: OS command line utilities, open source projects, commercial data mining and analysis products, resource monitoring utilities, and applications like Excel.  Being resourceful and combining tools is the key to quick and effective enterprise infrastructure analysis. 

I prefer the term infrastructure analysis because it avoids saying, “Is it the network or server?” When it comes to troubleshooting a problem, I look at everything starting from the way devices talk to each other (device and even user behavioral analysis based on protocol analysis) and digging in or drilling down from there.  Sometimes I get lucky and find a simple physical connection duplex mismatch causing CRC errors leading to TCP retransmissions. Other times it gets far more complex, like when I get into n-tier analysis with a multitude of devices, protocols, server types, and applications. But I digress.

Regarding the shark, as you probably guessed I’m referring to Wireshark (formerly Ethereal), updated this month to version 99.7.  I would not recommend depending solely on Wireshark unless you have time to burn doing stuff manually that could be done much faster using other tools. For that matter, some data you just can’t get with any protocol analyzer.

But the shark is in my tank for good reason.  I suppose one reason is that it’s free.  But time is money and free tools can actually be more expensive compared to shelling out a few (thousands) dollars for commercial offerings if the commercial products save you valuable time. But the shark is quite good for certain tasks.

One of the more obvious reasons to use Wireshark is for its breadth and depth of decodes.  Believe it or not, some analyzers actually have deeper decodes for a few protocols, but none have the breadth.  For example, in a recent analysis of a large enterprise, I needed decodes for Distributed Relational Database Architecture (DRDA), a standard driven by the Open Group and IBM for accessing distributed data based on SQL.  While the SQL statements could be read as plain text in the TCP payload, it helped to know in what context these commands were being used.  A couple of other unnamed commercial analyzers that I had access to did not decode it.  The shark to the rescue.

A not-so-obvious use for Wireshark is packet conversion—read in one format and write to another.  Observer native format to OmniPeek native format?  No problem.  Most analyzers support the simple Sniffer .enc format as well as their proprietary format.  The problem is that some do not do a good job of converting to .enc, resulting in packets that are flagged as “sliced” but really aren’t or have the occasional timestamp conversion problem.  Wireshark seems to do a pretty decent conversion to .enc, at least a lot better than one unnamed commercial offering.

The shark is also portable.  I’m not so much referring to its multi-OS portability as I am to its USB flash drive portability.  I can run a version of Wireshark right off a U3-compatible USB flash drive without having to install it on the host system.  A shark in your pocket.  How cool is that?

There are numerous other tricks for the shark that can complement your analysis including some cool command line stuff.  You can learn from the experts that live and breathe the shark at the forthcoming Sharkfest that promises to be more exciting than, well, the original Jaws classic.  The event is March 31st – April 2nd in Los Altos Hills, CA (near Sunnyvale). 

I can envision the feeding frenzy if the highly energetic Laura Chappell shows up dressed in a shark outfit with a big TCP FIN on her back.  Ouch.

Inside 802.11n Frame Aggregation

I’ve been busy lately working on a white paper entitled "Inside 802.11n Wireless LANs: Practical Insights and Analysis” now available for download. In the section ”Cutting the Overhead” I detail how 802.11n has improved the efficiency of the MAC protocol using a single block ACK (BA) for multiple frames as well as the ability to aggregate multiple frames into a single transmission.  I’ve excerpted information on frame aggregation in this blog.  For insight into the block ACK along with frame capture analysis, please consult the white paper.

802.11n can send multiple frames per single access to the medium by combining frames together into one larger frame.  There are two forms of frame aggregation: Aggregated Mac Service Data Unit (A-MSDU) and Aggregated Mac Protocol Data Unit (A-MPDU).

A-MSDU increases the maximum frame transmission size from 2,304 bytes to almost 8k bytes (7935 to be exact) while A-MPDU allows up to 64k bytes.

A-MSDU creates the larger frame by combining smaller frames with the same physical source and destination end points and traffic class (i.e. QoS) into one large frame with a common MAC header.  One way to visualize this is an access point receiving frames from the wired side at a rate faster than it can transmit them on the wireless side.  Ethernet frames headed for the same wireless client can be queued then combined into one larger frame for single transmission, cutting down the overhead dramatically.

The Caveat

One caveat, and it’s a big one, is that like any wireless transmission, the larger the frame the less likely it will be received with no errors.  I’ve observed that 802.11n senders tend to learn the maximum data rates possible for given frame sizes.  Thus for instance, you may see a frame containing a TCP ACK send at 270 Mbps (or 300 Mbps if a short guard interval is in use) because it’s very small (typical 78 bytes including the 802.11 overhead).  Frames containing FTP data may be sent at 121.5 Mbps simply because previous attempts at a higher data rate proved futile.

Thus it’s not clear how much we will gain in efficiency using the A-MSDU.  With its frame size up to 7935 bytes, with one MAC header and payload protected by one CRC like any other single frame, it will most likely be transmitted at a lower data rate for reliable reception.

Contrast this to the A-MPDU which is essentially a chain of individual 802.11 frames sent back-to-back with one access to the medium (i.e. one preamble).  The destination must still be one address and the traffic class (QoS) must be the same for each.  Clearly, there is more overhead with the A-MPDU because we still have individual PDU frame headers vs. one in the A-MSDU.  Unlike the A-MSDU however, individual PDU frames also have their own CRC; an error in one PDU will not affect the others in the group.

The bottom line is this:  Reliability has its overhead.  The A-MPDU allows a much larger "burst" of frame data to be sent compared the the A-MSDU.  In fact, many 802.11n access points conservatively option for the 1/2 A-MSDU frame size (3839 bytes) by default to play it safe.

802.11n Close to Final as TGn Draft 3.0 Approved

The IEEE P802.11 Task Group N (TGn), the overseeers of the 802.11n project, approved TGn Draft 3.0 in this week. According to the official status report fresh out of the Atlanta meetings, 240 out of 325 or 85% voted affirmative, surpassing the required 75% required to carry.  Several hundred comments, largely editorial and techical clarifications from working members since Draft 2.0, have been resolved.  A letter ballot will go out to approve TGn Draft 3.0 as an official 802.11n draft.

Further comments have already been generated by the vote on 3.0 which is typical.  According to the IEEE "Comment resolution will promptly continue in a series of authorized conference calls and a three day ad hoc session to precede the next interim meeting in Taipei."  FYI, the letter ballot on TGn Draft 2.0 that became an approved 802.11n Draft standard, generated approximlay 740 comments which eventually resulted in an approved TGn Draft 2.05.

It seems like an endless cycle but TGn is calling for a final 802.11n standard in October 2008.

The Route Less Traveled: Musings on Layer 3 Switching

According to Wikipedia, the only difference between layer 3 switching and routing is in the internal hardware implementation.  Layer 3 switching takes place in specialized hardware vs. general purpose microprocessors in traditional routers. Furthermore “Layer 3 switches can be placed anywhere in the network because they handle high-performance LAN traffic and can cost-effectively replace routers.”  Thus, the motivation behind layer 3 switching appears be a need for high performance routing in our LAN environments.

Traditionally we built our LAN (and WAN for that matter) infrastructure by dividing our IP address space into subnets, which required inter-subnet routing.  The escalation in LAN bandwidth per user and changing application performance requirements placed additional demand on routers.

But let’s not overlook one very important aspect that had a far greater impact:  Over the years we virtualized our LANs into VLANs.  The original motivation for doing this was to control broadcast domains as our layer 2 switching infrastructure grew.

Dividing our LANs up into VLANs led to new benefits in the areas of security, management, and user control to name a few.  We even dedicated VLANs to applications requiring different handling characteristics such as VoIP.  Some esoteric uses for VLANs included assigning multiple SSIDs to a wireless access point in order to map to VLANs on the wired side.  This allowed us to prioritize incoming wireless traffic, restricting uses to certain resources, or both (corporate vs. guests or faculty vs. students, for instance).  It seems that our imagination became the limit.

Unfortunately we were faced with a rather nasty problem - how to get traffic between these isolated broadcast domains?  After all, packets like ARP broadcasts were stopped cold in their tracks. The answer?  Associate a VLAN with a particular subnet and use a router behind a switch to handle VLAN-to-VLAN traffic.

How efficient!  Not.  Extra packet hops and traffic across LAN segments and significant additional load on the router ensued. This problem became affectionately known as "local routing" and all the expert systems in protocol analyzers detected it (well the good ones anyhow).  Two devices could be plugged in side-by-side in the same layer 2 switch but on different VLANs and subnets.  Instead of containing the traffic inside the switch with high-speed switching between ports, the packet was sent up to a router and back down again, wasting bandwidth and router resources.

So what came next?  We threw more backbone bandwidth and router CPU at it!  I'm only half serious here. But for a while it was the only way to solve the problem and some are still doing it this way today.  Layer 3 switching to the rescue.

By pushing layer 3 switching down to the device per port level and applying techniques similar to content addressable memory found on layer 2 switches, but for IP addresses instead of MAC addresses, we solved our little VLAN problem.  Not only that, but we can now have as many ports as we want dedicated to one VLAN/subnet to serve multiple devices.  No more restrictive one-to-one subnet-to-router port relationships.  Slick.

One thing I really like about layer 3 switches when it comes to network analysis and troubleshooting is that they decrement the TTL in the IP header, just like a conventional router.  Thus allows you can see how many store and forward devices a packet has traversed, even if they were virtual hops within a single layer 3 switch.  Try that in conventional, transparently-switched layer 2 environments.  You have to resort to manually figuring out the switch route or using specialized analysis tools to follow the switch MAC address tables via SNMP. Fluke Networks specializes in this sort of thing, as a matter of fact.

As for layer 3 switching vs. routing in your environment, much of it boils down to how many localized VLANs you have and how much traffic passes between them.  This becomes an interesting exercise in analysis because some applications like VoIP will have a dedicated VLAN up to a certain point (think about scalability) and yet other applications and servers will have their own unique switching and routing requirements.
 
Another interesting question is at what point do you push layer 3 switching down to the workgroup?  I’ll leave that as an exercise for the student.

A Company is Born: Bitcricket

We interrupt this blog for an important announcement.

I’m proud to announce the birth of Bitcricket, my new company with an aim to make the network analysis experience personal again (with apologies to HP’s “The PC is Personal Again” campaign.)  You, my blog readers, are getting this exclusive before it hits the wire.

To celebrate, Bitcricket is giving away the brand new IP Subnet Calculator for IPv4 and IPv6.  With the Federal IPv6 mandate right around the corner in 2008, what better timing?  You can play with the various IPv6 addressing schemes, beginning with a choice of IPv6 addresses already configured in your workstation (the default for Vista and MacOS X).  Meanwhile, your favorite IPv4 mask, subnet, and CIDR features are there too.  The calculator has been designed to be cross-platform; it runs natively on Windows and MacOS X (both PPC and Intel architectures).

If you’d like, I can come on-site and teach real world protocol and network analysis with the tools you already own.  I am going to assume that you are already familiar with their basic operation via the owner’s manuals or introductory webinars from your vendors.   Likewise, I will gladly work side-by-side with you on troubleshooting and optimizing your most challenging application and/or network performance issues.

Happy troubleshooting!

Thank you and back to our regularly scheduled blog.

The Ups and Downs of Network Technology Stocks

I confess that I’m not a serious day trader although I do keep one eye on the market, especially the networking sector.  I’m always curious to see how the more focused companies like a NetScout or Cisco perform vs. umbrella technology companies like an IBM or HP.  So here’s my foolish attempt to write a financial blog. It’s  not quite what you’d expect from Motley, but you may find it interesting if you’re not exactly a market guru.

A great way to do a little stock trending analysis is via Google Finance.  Just type in a company name or ticker and a nifty history chart appears. It allows you to slide a window back and forth to select a period of time, move your mouse along the line chart to display the exact date, and see the trade volume bar chart at the bottom.  I think the coolest feature is the attachment of tags along the graph that reference financially related news at those exact dates.  If you’ve never tried it, give it a shot and type in NTCT for NetScout and follow along.  The figure below is the chart as of November 4th.

If you had put $2000 into NetScout back in May, it would be worth about $4000 today.  Not a bad return for less than 6 months time.  Sure beats my bank’s 4.872% (I made that up) CD for the same time period.  Just for fun, let’s follow the chart along with the recent news, as indicated by the letters.

Letter I, May 3 – Revenues are up but net income is below analysts’ estimates by a couple of pennies.  Oh rats, stock tanks shortly thereafter to $7.25.

Letter H, June 25 – The stock is now valued at $8.75 after a slow, upward climb of a buck and a half.  Fifty cents of that increase came the day NetScout announces a “Raised Q1 Guidance.”

Letter G, July 25 –A mere month later a similar Q2 Guidance (that was a rather short quarter, no?) is issued adding another million to the revenue projection and the stock knee jerks to ten bucks before settling down into the low nines the remainder of the summer.

Letters C-F, September 20 – The Network General  acquisition announcement generates some positive interest, kicking the stock up another buck or so.

Since then, it seems we’ve had lots of time to noodle the situation.  The most recent news last week is that the acquisition is officially complete.  My sources tell me that we’ll have to wait for the first round of real integration until April 1st of next year.  Meanwhile, it’s status quo.
A number of people I’ve spoken to thought the deal  was steal for NetScout at only $50m in real money, i.e. cash.  Since then, the stock has see-sawed upward, peaking at nearly $15.50 on Halloween.  Downright scary.

As an exercise to the student, try the same for Cisco and note how you could have doubled your investment albeit over a slightly longer period, one year.  Still beats that CD.

Meanwhile, those that bought NetScout in the weeks prior to the announcement are laughing all the way to the bank.  I wish I was one of them.

Network Performance Management – Winners and Losers

Network Physics, a company that venture capitalists invested some $55m (click here for the history), essentially liquidated this week to OPNET for a basement bargain $10m.

I have fond memories of Network Physics at Interop, with guys and gals running around in those loud “It’s Not the Network!” t-shirts.  The premise was that their packet-based NetSensory appliance monitors application flows on the network and distinguishes server vs. network delay.  I also worked with one of their brilliant engineers on the Apdex board of directors.

It looks like a loss for Network Physics (certainly the investors) but a win for OPNET. Near as I can tell, OPNET will ditch further development in their own their performance management product, ACE Live (announced just last month), and instead go with NetSensory and rename it ACE Live.  Got it?

There are many players in the network performance management space with overlapping areas and niches.  For instance, Coradiant focuses on Web based performance analytics.  NetQoS focuses on Application Performance Management although they are beginning to broaden.  They also have some nice integration with Cisco WAAS (Wide Area Application Services).  Larger umbrella companies like Compuware and Computer Associates, also have tools that watch application performance.  Compuware’s ApplicationVantage comes to mind (part of their Application Assurance portfolio) and CA has tools in this area focused on SLA’s in the web space.

As with any acquisition, how well will the acquiring technologies integrate into a much larger beast?  Track records have been mixed.  Some 25% of Cisco acquisitions don’t pan out.  Witness Network General’s failed comeback and acquisitions and now it’s NetScout’s problem.

Will larger companies lose their identity in this market, specifically application performance management/end-user satisfaction within the network performance space?   Or will smaller, but significant and fast growing players like NetQoS trump the big guys?
 
Speaking of NetQoS, how did they get so far ahead of Network Physics in the performance appliance biz?  I’d say partly that Network Physics suffered when shifting gears (like changing your college major mid-term) from shaping traffic to measuring the stuff and that NetQoS recognized early on the value of a feature rich (as in superb GUI) browser-based console coupled with a smart appliance, but I digress.

One thing I do know is that all the long and hard work over the years from Peter Sevcik (the NetForecast guy that started Apdex) pushing the importance of measuring the user-experience, is coming front and center.