As the go-to source for all news relating to bufferbloat, I’m glad to announce that the first of several possible solutions to the problem will shortly be available, just in time to save the Internet from self-destruction.
What, you didn’t know the Internet was self-destructing? Well it is.
Bufferbloat, my #1 prediction from 2011, is an artifact of cheap memory and bad planning in the Internet Age. In order to keep our porn streaming without interruption we add large memory buffers in applications, network cards or chipsets, routers, more routers, and even more routers until the basic flow control techniques of the TCP protocol are completely overwhelmed. Data glugs through the system like a gas can with no vent. Our solution to date has been to make our pipes (and therefore our glugs) bigger, but in the long run that won’t help. Latency increases and performance declines.
Many Internet users are unaware of bufferbloat because it has been masked by faster computers and bigger pipes and because it sneaked up on us slowly over time. But here’s a test. Think back to your first broadband cable or DSL Internet connection, right after you finally got rid of dial-up. How much faster is your Internet connection today than it was back then? Don’t think in terms of numbers but of subjective performance. It’s not much faster at all, is it? That’s bufferbloat.
My first broadband connection was a Northpoint DSL line back in 1996 running at 384 kilobits-per-second and my primary computer at the time was running at something like 40 MHz. My Internet connection today is 24 megabits-per-second (about 70 times as fast) and my main PC has quad cores running at three GHz (500 times as fast!). My Internet connection should certainly feel at least 70 times faster. Yet for most purposes my connection doesn’t seem any quicker than it did back in 1996. Part of this is that we are pushing more bits through the pipe, but the rest is bufferbloat.
The solution to bufferbloat is Active Queue Management (AQM), which is controlling buffers to maximize data throughput. The most effective thing most of us can do right now to reduce bufferbloat is to manually set all our buffers at the smallest possible size, but that’s just a coping strategy, not AQM. AQM will dynamically adjust buffers to improve network efficiency with the result that our Internet connections will all speed up as if by magic.
But as it was with Tinkerbell, such network health can happen only if we all believe. To eliminate bufferbloat it isn’t enough for you and me to change our ways, we all have to do it.
And we finally are getting the tools to do so. This morning the first public bufferbloat solution appeared in a paper on an ACM website titled Controlling Queue Delay by Kathleen Nichols and Van Jacobson. If you read the paper please also look at today’s blog post from bufferbloat pioneer Jim Gettys explaining what it’s all about.
But the best is yet to come because within days or weeks we’re likely to see AQM-equipped beta code for Open Source routers from many manufacturers. Depending on how well this code works and how quickly manufacturers like Cisco, Netgear and others adapt it for their non-Open Source network hardware, this could be the beginning of the end for bufferbloat.
It’s nice to print good news for a change.
Of course I sometimes get carried away, so here’s a cautionary note from Jim Gettys: “We have only ethernet running. Wireless will be *much* harder, and take months to years to get working properly. This has to do both with the much more complex queuing needed for AP’s, and the hair that packet aggregation causes (802.11n). Also CeroWrt only runs on a small amount of hardware at the moment; and CoDel only on its ethernet so far. Also lots of other things besides home routers need fixing: e.g.
your cable/dsl modems, FIOS gear, etc…”
Like he said.
My advice to Cisco, Netgear, D-Link and others is that this could be an important moment in their businesses if they choose to approach it correctly. It’s a chance to get all of us to buy new routers, perhaps new everything. Think of the music industry bonanza when we all shifted our record libraries from vinyl to CDs. It could be the same for networking equipment. But for that to happen the vendors have to finally acknowledge bufferbloat and use their marketing dollars to teach us all why we should upgrade ASAP. Everybody would win.
Take our money, please.
You Go, Tomato USB.
Or they could show some good will and encourage people to update to newer firmware for their existing supported equipment (at least anything < 5 years old) so as to overcome the issue.
Modems (cable, DSL, fibre, etc.) will be upgraded simply because it is in the ISPs interest to do so, at least when they own the hardware.
Bob is absolutely right. It’s a great opportunity for the infrastructure vendors to improve their products, sell and make money. Good thing too since things were getting pretty hum-drum and commoditized for them lately.
If we all (or a vast majority) need to upgrade or it won’t work, then that’s equivalent to saying it won’t work.
Consider how many people run old insecure browsers and operating systems. Even when upgrading is free, they don’t. Add to that observation the tiny fraction of the population that has even heard of this bufferbloat problem, and of them, how many really care (or even believe it’s a problem), and of them, how many are willing to spend money.
Doesn’t look good.
However, if it’s the backbone and ISPs that only need to update, it seems a bit more likely, if it’s not too expensive.
upgrade options disappear after 2 or 3 years. The Market has decreed that we, the sheeple, shall all toss our old crap out every couple years and buy all new shiny.
what, you didn’t get the memo?
I have the tendency, for some reason, of dumping the OS provider when they dump my OS, since they are not offering me upgrades due to old chips, old video, not enough memory, brown shoes and navy pants, yo’momma, what the eff ever. man-critical equipment in the space industry lasts 25 to 35 years, folks. you boast about it there.
screw you for not keeping the update pipeline going. and it’s no better in the ISPs, believe me, I have a Tier 1 upstairs. you think they’re going to fiddle with buffer upgrades when they have to work through IPv6, I have a great deal on a bridge for you.
The backbones and ISP’s are all getting ready to update for IPv6 in silicon/ASIC anyway. If their IPv6 investment gets made right before bufferbloat-sensitive gear is available, that’ll hurt. I’m making the assumption the silicon/ASIC can be optimized to help with bufferbloat.
But, that aside, so long as they all do update, then users will update as their gear dies out, they upgrade for some other reason (e.g. new wireless standard) or they visit a friend and say, “holy cow, your computer is fast – when they get it explained that it’s not the computer it’s the Awesomeco BT-1337 router that made it so fast. They’ll probably never learn about bufferbloat, but will be happy to plunk down $40 to make their $45/mo Internet service much faster.
Better queue management is needed where the queues build, i.e. at bottlenecks. Typically this will be in last mile (DSL or cable) or wireless, not the backbone.
If my experience is representative, the average consumer router craps out in 9-12 months, tops, and this is a non-issue. I have always assumed defective hardware and short warranties to be the business model.
I guess my four year old Linksys WRT54GL is better than average.
Site response time is the same as when I had an ISP in 1996 using 56K modems. It seems the content providers will fill the river with enough cargo to keep site response times barely tolerable as they were in 1996 and are in 2014. Bufferbloat will not solve the site response problem, any more than a new lane on a freeway will ease traffic jams, given the inevitable but short lag time between lane completion and re-saturation. I suspect until consumers rebel by canceling subscriptions, site response time will remain nearly intolerable, and I don’t expect that to happen anytime soon.
Wow – the ‘Related posts’ actually relate this time! All 3 of them! Whooo!!!
That was pure luck.
Well sure, but good luck in this case. That’s worth something.
The VC’s tax treatment problem is a bit of a stretch. But yes, government meddling for it’s own benefit doesn’t help anything.
Better still, the first comment isn’t spam.
I take that back.
“…use their marketing dollars to teach us all why we should upgrade ASAP…” I’m afraid they have been doing that for some time. All they say is upgrade your speed and buy our newest router now (not after we fix the problems). What they haven’t been admitting is the the newer equipment may be no better than the old. In fact, thanks to the WPS fiasco and increasing buffer sizes, they could be worse.
I don’t get “…it isn’t enough for you and me to change our ways, we /all/ have to do it.”
If Mrs Eileen Krebshaw of Boise, Idaho is using unreasonably large buffers, how is that going to have any impact on me? If it results in higher latency for her, doesn’t that disadvantage her and benefit everyone else?
Surely the people who need to upgrade are the ones either producing bits or moving them from one place to another. Since you could expect that the more bits involved, the more likely they are to upgrade, this should sort itself out fairly quickly.
Bob, I think something’s hosed with your RSS feed. All the links to your articles look like this:
http://feed/?p=3927&utm_source=rss&utm_medium=rss&utm_campaign=watch-out-ibm
And when I click such a link, I get this URL instead:
http://landing.trafficz.com/index.php?domain=feed.com
Just thought you should know
Does the Twitter feed’s tinyurl get generated from the RSS feed? Similar breakage there.
I’m getting these feeds today. The first is OK. The second shows XML code instead of Cringely Comments. The third has only mp3 files which ended in July of last year:
http://feeds.feedburner.com/ICringely
http://feeds.feedburner.com/ICringelyComments
https://www.cringely.com/feed/podcast/
Bob,
The VoIP Users Conference, a weekly podcast that I co-produce, had a good session on buffer bloat a few months ago. Anyone interested in the topic can find the recording here:
https://www.voipusersconference.org/2011/bufferbloat/
It’s particularly of concern to anyone who has an interest in real-time communication over the internet. That includes voip in it’s present common forms, but also emerging technologies like WebRTC.
Michael
Hi Bob,
Usually you’re an OK chap, honest, I like reading your stuff. But with regard to bufferbloat, on January 2011 you told us about “this huge mess for ourselves that we’ll all be talking about by the end of the year”. Well, a year and a half has passed, and while I agree bufferbloat is not a good thing, but I’m not sure about a “huge mess”.
“Huge mess” is a special term I usually keep for the Middle East and for IE6, not for stuff that I hardly hear about outside of from you, that only has 85K hits on Google, and that has a “beginning of the end” 18 months after you breathlessly told us how huge it will be. Hrmf.
I’d mark that prediction zero, dude.
– Yaniv
Bob, speaking of breaking the Internet, your RSS feed has been broken by Google…
When I clicked the link in my RSS feed I landed on this (Larry) page:
https://accounts.google.com/ServiceLogin?service=feedburner&continue=http%3A%2F%2Ffeedburner.google.com%2Ffb%2Fa%2Fmyfeeds
It looks the the “Plusification” of all Google is upon us, and Larry’s taking you and your readers for a ride. You might want to drop Feedburner.
“Part of this is that we are pushing more bits through the pipe, but the rest is bufferbloat.” But part of it IS that we are pushing more bits through the pipe, and not all of those bits contribute to the user experience I want. For example, I turn on the PC, and bring up Bob’s site. Then I look at the router – and I have 40 (!) TCP and UDP sessions going. Why? Good question. A lot of it is software “phoning home” to report on my use, check digital rights, check for updates, etc. Some of it is ads, animation, and other distractions. I expect very little of it is the content of the page that I wish to see.
Another part is that we get pretty good results and we’re hitting a point where differences aren’t noticeable on small screens. You can still see differences when doing full screen video, but many users don’t. There are still some points where the dreck on YouTube hangs or Netflix has oversubscribed again, but when the video is flowing most users can’t see what they’re missing. Assuming we’re going to see the deltas in network speed can be related to assuming we’ll see the differences between 1.0Ghz and 3.0Ghz processors: sure you might spot something, and fewer things will bog it down, but in most places most users won’t see much.
Use this: http://winhelp2002.mvps.org/hosts.htm . I just copy the text of the HOSTS file they provide in the link: “To view the HOSTS file in plain text form” and use that to replace the contents of the Windows HOSTS file using Notepad. Stops a lot of bad stuff. Also, it helps to have a way to toggle it on and off to see what it’s blocking so they provide this link: http://winhelp2002.mvps.org/hostsfaq.htm#Rename.
Yaniv
If those with arcane knowledge, special tools and unique opportunitues for insight are saying something is a problem, it often deserves special credence even if it’s a minority view, not apparent, nor self-evident.
Germ theory? Bah!
Too cold to launch the shuttle?! So you said last time (yet it was fine)!
Real-time applications (think VoIP) over 802.11n Wi-Fi is a great place to observe buffer bloat. You can sometimes observe buffer induced latencies you can measure with a wristwatch, in contrast to microseconds. The phenomenon is usually manifested under load. In other words, don’t think ping times; think ping times while maxing out your link
I like the idea that Cisco, Netgear, D-Link could ride in as knights in shining armor to rescue us from the problems they helped to create.
Funnily enough it was 1996 I first got Internet at all 😀 – down the phone cable using a screeching 14.4k serial-port modem. I’ve also got a couple of 640×480 screen-grabs of the AOL browser back then and it’s a high percentage of actual content – necessitated by the low refresh rate and small screen real-estate. Ads way more insignificant than nowadays.
So I wonder if the 2012 buffer bloat is an end-users fault at all. Surely we are going to be the ones who pay the industry’s way out of it. Ok, I concede there are millions who now watch movies over the Internet, something I would have thought a neat idea back in 1996 but dismissed as totally unrealistic of course! :-))
My present broadband router is about 6 or 7 years old. I bought it as a reconditioned item, but a good quality one. Still goes strong, I’d buy a new one if it helped buffer bloat AND had a few extra features like MAC address time limits. That neat idea would enable me to keep Internet going all day but allow me to limit my son’s to a time frame of say 9am to 9pm.
It’s that kind of functionality that might easily tip the balance for me to upgrade – and then help defeat the evil of Buffer Bloat!!
;-))
If you care about buffer bloat you will use Linux which added some anti buffer bloat features a while ago.
Michael Graves posted a link above to a 90 min interview with one of the experts in the voip field. In it he mentions that Microsoft has managed to optimize the buffering for the case of 100 mb/s connections vs Mac and Linux which do not. He said “Windows at 100 mb/s showed no problems whatsoever” because they “very cleverly engineered around the problem for the 100 megabit network case”. These statements are made at the 32 minute mark.
I’m more concerned with personal bloat, and I doubt any fancy-dan mathematical equation is going to solve the problem.
But, I suppose, one fewer “bloats” is a good thing. Good job. Keep up the good work. Carry on.
Be seeing you.
“It’s a chance to get all of us to buy new routers, perhaps new everything. Think of the music industry bonanza when we all shifted our record libraries from vinyl to CDs.”
Firstly “all” means everyone and everyone wasn’t that stupid! Don’t assume everyone is like you.
Secondly never waste resources buying new when you can patch up existing things. The planet does not exist just so that some people can spend their time playing silly games of shop keepers! The rare elements that go into most electronics will be gone soon enough.
Don’t be one of those dumb people who always have to the latest thing. That is just being suckered by manufacturers who don’t know how to shrink gracefully into replacement mode once they have saturated a market.
“Think back to your first broadband cable or DSL Internet connection, right after you finally got rid of dial-up. How much faster is your Internet connection today than it was back then? Don’t think in terms of numbers but of subjective performance. It’s not much faster at all, is it?”
Back then it was postage-stamp sized grainy Youtube videos. Now I stream 1080P video with 6 channels of audio. So actually it seems much, much faster today.
Best spam bot ever
The REAL cure is to wake the F’ up and get rid of ALL old, outdated, tehory based copper core medium for the Tx of internet. I get 100 Mbps DL and 100 Mbps UL. Iv not had to buffer a video in quite a while. Not only that, Im a gamer who expereinces no lag, whnile others wtill stuck on the old copper method are complaining about nothing more than this lag they get.
The REAL cure is to come into our technology, as far as Intenet Bx is concerned. Fiber Optic is the way to go. I no longer get distracted while getting online because its like I have the internet on my HDD. It actually made a drop in my power consumption… and isnt todays times all about using less power but getting more done.
I didn’t know there was fiber to the home on the west coast. Where are you located? Last I heard, Verizon provided it in a few places back east and said it would not expand any further. What is your data cap?
Last time, we have discussed about the Yiwu agent and then Yiwu export agent on ywagent.com, and now we are going to talk about the benefits or advantage or procedure of Yiwu agent in our country.
As I have explained afore broad bazaar if the Yiwu agent is getting advised as an all-embracing abode of business with its array of articles getting exported worldwide. The absolute bazaar is broadcast over altered arena according to the affectionate of articles getting exported. Toys and apparel, handicraft, stationary, jeweler, cloths, accessories are the specialty of the market. Every year there is a growing no. of adopted companies and buyers.
Added and added inventors getting admiring and because of its different articles accustomed common by 200 nations. Yiwu has become the admired abode for abounding humans who wish to alpha their own broad business from china. The government has fabricated the rules simple for adopted investors which accomplish them force to advance added in Yiwu market. There are lots of cable companies which accommodate able Yiwu abettor to accomplish trading from bazaar simple and fast. Let me explain you the accessories and casework Yiwu agent abettor can accommodate you in authoritative acknowledged business deal.
Why not to try theYiwu agentto improve the efficiency of your business? Don’t be hesitated and you will find how great the agent is for your business!
Apparently its better to upgrade the internet to have a better communication and to avoid lag. At first I hesitate in upgrading the system for it’s another expense but I realized that having a poor internet connection cost a lot and it enables my blood raise high so I decided to upgrade and Boom! I am now enjoying internet and able’s to watch movies without interruption at the right cost.
computer…
[…]Beginning of the end for bufferbloat[…]…