This is my promised update on bufferbloat, the problem I write about occasionally involving networks and applications that try to improve the flow of streaming data, especially video data, over the Internet but actually do the opposite, defeating TCP/IP’s own flow control code that would do the job much better if only it were allowed to. I first mentioned bufferbloat in January 2011 and it is still with us but the prognosis is improving, though it will probably take years to be fully resolved.

If you read my last column on LagBuster, you know it’s a hardware-based workaround for some aspects of bufferbloat aimed especially at gamers. LagBuster is a coping strategy for one type of bufferbloat that afflicts a population of people who aren’t willing to wait for a systemic cure. LagBuster works for gamers and might be a workaround for other kinds of low-latency data, but that’s still to be determined.

One thing we learned from that column was that bufferbloat isn’t peculiar to routers: where there are separate broadband modems those are affected too. Here’s a very telling comment from LagBuster co-inventor Ed DeWath:

“The problem isn’t the router. The problem is in the modem.

Let’s use your example – you have a fantastic router with DD-WRT and the best QoS in the world. Your router detects a high #1 priority packet and places it at the top of the egress queue so that the high priority packet immediately exits the router at the “front of the line.” Wonderful! Next, the high priority packet immediately enters the “end of the line” of the modem’s single adaptive rate buffer (ARB). Well, modems have no idea about QoS or prioritization, so your high priority packet must wait in queue behind ALL the previous packets before it exits to the Internet. All packets must wait for their turn — there is no jumping to the front of the line in a modem! Put another way, modems remove any semblance of “prioritization” from packets – all packets are equal in a modem. Democracy in action!

That said, the LagBuster solves the modem lag problem by precisely matching the ingress and egress flow rates, and limiting the modem to only one packet in the ARB at any time. Further, with LagBuster’s dual buffer, the high packet priority status is retained. With no other packets in the ARB to cause queuing delays, the LagBuster can always send high priority packets from the high priority buffer at maximum speed.

Lastly, the dwell time (“lag”) in the modem is basically proportional to the modem’s buffer size and upstream bandwidth. A consumer grade modem may add up to hundreds of milliseconds of delay to the packet stream. For example, a typical modem buffer is about 300KByte, and a typical upstream speed is about 5Mbps, which can result in as much as 500ms delay for a packet flow. QoS routers cannot ever solve that delay. LagBuster does.”

A lot of information there that the smartypants contingent in this audience can argue about all day (“But my modem is combined with my router” — Really? Is it logically integrated or just living in the same plastic box? How do we even know?) but it is very significant to me that there are bufferbloat issues peculiar to broadband modems. I talk with the best bufferbloat experts in the world and some of them hadn’t devoted a single neuron to the modem buffer, yet from what Ed says, above, ignoring the modem could hobble an overall bufferbloat solution.

I’ll detail below what’s happening to cure bufferbloat, especially in the Linux community, but first I’d like to do something uncharacteristic, which is to make a call to action to a couple specific vendors.

The golden age of the music business (note my italics, I’m not calling it the golden age of music, but a great time to make a lot of money) was in the 1980s and 1990s when we all converted our vinyl record collections to CDs, paying anew for music we already owned.  People who sell stuff love it when a technical change requires we get all new stuff. The advent of digital television, for example, sparked a boom in flat screen TVs that is only now turning into a busted bubble, but not until a lot of money was made.

Bufferbloat offers just such an advantage quite specifically to Cisco Systems and to the Motorola Mobility division of Google. Both companies make cable and DSL modems as well as routers. Other companies make these, too, but Cisco and Google can claim bigger network infrastructure shadows than any of those other companies. Cisco dominates the service provider end of the business while Motorola and Google have the most influence on the consumer side, even more than Microsoft or Apple.

Each of these companies should love to sell us all over again a hardware/software solution that eliminates bufferbloat and makes our networks sing. There is no reason why every broadband connection in America can’t have 100 millisecond latency or less. There is no good reason why this can’t be implemented in a year or two rather than 5-10 if these companies will make doing so their marketing priorities.

Everybody wants this and everyone who understands the issues seems willing to pay a bit to make bufferbloat go away. That means it is a terrific commercial opportunity, yet I don’t see much action. Some of this comes down to proprietary vendors not wanting to expose what they are doing, but much of it comes down to misplaced priorities or simple ignorance on the part of industry executives. I wonder if John Chambers at Cisco or Larry Page at Google have even heard of bufferbloat?

This is not to say that Google is doing nothing about bufferbloat. Current state of the art in defeating bufferbloat is an Open Source project called the CoDel AQM algorithm originally by Kathie Nichols and Van Jacobson, but the lead developer (with at least five others) Eric Dumazet has just joined Google. Dumazet, who is French, is responsible for codel, fq_codel, and TCP Small Queues if you want to go poking around.

CoDel is now an official part of Linux.

Linksys, after a long period of stagnation and outsourcing, is in the process of being more brought in-house. Cisco has contributed toward analyzing the behavior of several revisions of the codel algorithm, I’m told, but is apparently still unconvinced that codel is the bufferbloat solution.

Apple is reportedly well along in its own bufferbloat solution but, as usual, there is no real news escaping from Infinite Loop.

For those who want to reflash their routers and experiment, Cerowrt 3.3.8-27 “sugarland” runs some advanced forms of codel on all interfaces including WiFi, which is the area where the most work still has to be done.

In the oddest yet also most encouraging bit of news, codel over WiFi is right now being tested on a special network built where no other WiFi networks impinged — in this case at a clothing-optional resort in Northern California, of course.

“Still, tons of work do remain on everything,” reports one of the guys in the thick of it, whose name I am witholding to protect his innocence. “Head ends and cable modems are particularly dark to us, the home gateway vendors are asleep at the switch, ISPs clueless, pipelines long, and the biggest problem is that ALL the chipset vendors for Customer Premises Equipment (CPE) have moved the fast path for IP into hardware, rather than software, so it’s going to be hard to change course in the next three years unless some disruptive CPE maker shows up and leverages an ARM chip…”

Hint hint.