Remember Bufferbloat? It’s a subject I was among the first to write about a decade ago, starting with a prediction column just like this one in 2011. The problem at the time was that every video or audio application — the big bandwidth consumers — was trying to solve performance issues through pre-buffering. You’d launch Netflix (just one example — they all did it) and it would pause for a few seconds filling a huge buffer intended to smooth-out any playing glitches. Except performance didn’t improve and in fact got worse because of buffers buffering buffers. These extra buffers were defeating TCP/IP’s own flow control mechanisms, often leading to total failure of the connection. Jim Gettys from Bell Labs called it Bufferbloat, then Jim and Dave Taht spent the next three years or so fixing the problem, or so they thought.
Well Bufferbloat is back.
Fair queuing and Active Queue Management (AQM) in home routers “solved” Bufferbloat in 2012, though only if people upgraded their router firmware. Many users still haven’t upgraded. Smart Queue Management (SQM) did become widespread. And, most important, fq_codel (RFC8290) was built into Linux, BSD Unix, iOS, and OS X — but not into the core ISP bottleneck links (Cisco and Juniper) where it mattered the most. AND IT WASN’T BUILT INTO WINDOWS, EITHER.
So Bufferbloat got better, just not all better.
Today hundreds of aftermarket products are providing really good de-bloated performance in the range below 300 megabits-per-second. They help a lot on connections of 250 megabits or less.
In 2021, with most of us working from home as citizens of COVIDville, our use of video conferencing and video entertainment apps has increased dramatically and people are running into Bufferbloat again. Complain to your ISP and they’ll advise a bandwidth increase to solve the problem for more money. Except it won’t solve anything.
If you are working at home and trying to co-exist with your kids, DON’T buy more bandwidth, because it only makes the problem worse. Instead, keep your bandwidth to 250 megabits or less and get a router that runs the right firmware. Your home network — and your bank account — will thank you.
I dropped my Xfinity connection from 400 mbps down to 250, saving some money. I have an Eero mesh network with five routers in all and Eero has a pretty good Bufferbloat solution so I am good to go.
But here is something important: one reason my Eero system is good is because my routers are all older units. More on this in a moment. But first I need to yell at ISPs.
All of the anti-bufferbloat internet standards mentioned here have been published and available for at least eight years, yet hardly any large ISPs support the new code in their default equipment. I can only guess that’s because it solves too many problems and lessens demand for bandwidth. Think about it, ISPs make a killing selling you excess bandwidth you don’t actually need. Electrons you aren’t using don’t exist and therefore those bandwidth upgrades are 100 percent profit.
Your ISP would rather upsell you — actually decreasing performance — than fix your problem.
Your best solution for distance learning is to buy a connection that is 250 megabits or less and buy (don’t rent from your ISP) a third-party router, making sure it is RFC8290 compliant. Older Eero and Ubiquiti routers are good, as is pretty much any router that runs the latest version of OpenWRT or DD-WRT firmware.
Google WiFi, and especially Nest, are terrible. Plume is terrible. If you are a StarLink beta user, they missed the boat, too (thanks for nothing, Elon). These companies are just a firmware upgrade away from being okay, but they have to recognize the problem and fix it.
EvenRoute V3 and Ubiquiti’s EdgeRouter X are my top two picks for economy routers that work fine right out of the box.
If you don’t know if you have bufferbloat, you are lucky. Check by running the DSLreports speed test, NOT the speed test from your ISP.
The big bufferbloat problem now is in new gear targeted at speeds most Internet users don’t need and can never achieve, with binary blobs provided by crabby makers of router chipsets unwilling to let anyone touch their code.
As for WiFi 6 and Orthogonal Frequency-Division Multiple Access (OFDMA) — both the core and overhyped technologies required to get it to scale decently past a few dozen clients — they don’t really work. MU-MIMO never got around to working either. Even for Eero, sorry.
The only present advantage of WiFi 6 products is more spectrum. Otherwise you have vendor lock-in combined with bad performance and poor reliability. Users are much better-off staying for now with 802.11n or ac at the mid tier of Internet speeds with good bufferbloat fighting tech if they want to sit at home sharing the link with their kids.
Maybe this will be the year it is widely recognized that bad videoconferencing can be traced to bufferbloat, and easily solved by users re-flashing their routers to OpenWRT or DD-WRT.
Maybe the FCC will finally get on the stick and at least measure the problem.
Or maybe not.
Yay! Nice move, Bob. You might have hit the nail on the head.
Thankyou… just updated the dd-wrt and enabled fq_codel
I think your statement about Google Fiber is incorrect. My ISP in Austin, TX, is Google Fiber. DSLReports gives me the following grades:
– Overall A+
– Bufferbloat A
– Quality A+
Upload and download speeds exceed 500 megabits/second.
Oops, never mind. My previous comment reported results when I was connected via Ethernet. On WiFi I get the following:
Overall – B
Bufferbloat – C
Quality – A
Speeds are less, around 300 megabits/second.
Looks like you are right after all. Sorry about that.
God what a load of crap. Unusable template and a rehash. These “predictions” are getting worse. Does Mark Stephens (aka Cringely) have alzheimer’s?
Let’s ask him, then. Trash talk – do you have Alzheimer’s? Alright, now we’ll see if he responds now that we’ve asked him.
He has a bad case of washed-up hack.
Bufferbloat can’t come back if it never left. I’ve heard 802.11ac described as being designed to do Bufferbloat.
The slippery thing about Bufferbloat is that its impact migrates to wherever the bottleneck is. If the WiFi is not the bottleneck, then the bloated buffer in the WiFi chipset will not cause problems. If the WiFi side is the bottleneck, then the bloat in the WiFi chipset is what’s killing your connection.
I heard that Comcast did implement active queue management on the cable side of their routers. Downgrading your connection to 250 Mbps works because it moves the bottleneck back to the cable side of the router. I would prefer to upgrade the WiFi.
Personally, I do have use for bandwidth over 500 Mbps, because the pandemic forced me into video editing. I don’t like Ubiquiti routers. The annoying thing is that no MIPS or ARM-based router I know is fast enough to do active queue management over 500 Mbps. I probably should switch to a PC as the router.
Who cares about “bufferboat”. It’s just Mark Stephens (aka Cringely” boasting about what a visionary he is and being whiney whiney about big corporations and rewriting everything so he is the centre of attention while crowding out everyone who has anything useful to say. (“It’s in the pipes. It’s in the pipes maaaaaaan” like some Frankenstein report of Aliens 2 and Three Mile Island”)
Yes everyone has more bandwidth than they know what to do with if they can pay. My local cable provider can provide 1GB if you have stupid money. OMG OMG Bufferbloat OMG. Well, duh. Not every part of the internet whether network or server or software etcetera is going to keep up. Who cares? Deal with it. This latest pediction isn’t a prediction at all. It’s just rewarned reactivity months (years) old news and typically US-centric while we are at it. Yeah the US internet is so shit Google just put their own US-UK pipeline in and, yes, it went through the UK because it’s the fastest connection for mainland Europe. Mainland Europe has its own issues. I doubt anyone would want to route an entry point via Bulgaria but OMG OMG Bufferbloat. Christ, give it a rest already.
Also Mark Stephens (aka Cringely) can’t even hit his own deadline for Mineserver news. “It’s in flux. It’s in flux maaaaaan.” Because that’s a BIG BOY word he heard some VC say once and Mark Stephens is trying to be a wannabe Musk like he was a wannabe Bill gates and a wannabe Steve Jobs etctera.
@trashtalk it’s spelled “et cetera”, “etc.” or “&c.”
@Ellery Mitchell
Depends on your usage and dictionary. If that’s the best distracting nitpick you can come up with don’t waste peoples time. I’m not having a random internet wanker on an American blog tell me what my use of English should or shouldn’t be.
“people’s”
Fios with an older Fios Wi-Fi router (4th gen):
Overall – A+
Bufferbloat – A
Quality – A+
This is with their “gigabit” plan. 430 Mbps download over Wi-Fi, 950 Mbps to the optical terminal. I don’t need that speed but last time I checked to downgrade, lower speeds were the same price or higher, in part due to a promotional discount that has no termination date.
Fios gigabit with a home made pfsense firewall/router:
Overall – A+
Bufferbloat – A+
Quality – A+
Another issue is that if you have a functional Wi-Fi environment and are working with older WiFi devices to exchange data, listen to music etc. then if you, or your neighbor, adds a Wi-Fi 6 router it seems to interfere with the old units Wi-Fi transmissions. These days the Wi-Fi spectrum is crowded, adding “reliable” routers is making it worse.
The articles that are advertised on my Google page are getting worse. No I don’t remember bufferbloat and judging by the description I certainly don’t get it now and have rarely if ever got it on the past. Im guessing this isn’t a UK thing as in the main our ISP are pretty reliable even with the bundled routers. Given the nature of how Netflix works its probably the last thing on earth that will buffer of any description. Forgive me but I’ve never heard that downgrading your package would actually speed things up, personally I think that is a complete load of shite, as is buying an older router unless it’s an upgraded older model
@Adrianos
I cannot comment on other ISPs in the UK but know VM hosts Netflix caches on their network.
I couldn’t give a stuff about 99% of the US side of the internet as I never see it.
As you frame it here, bufferbloat (like everything important) is a collective action problem. And what the past twelve months have shown is that collective action is no longer possible, toward any goal. If it depends upon collective action — and everything worthwhile does — it is absolutely not going to happen.
Once you are done with this year’s predictions(*), look a little further out and talk about what is going to happen in 20 years when the pipeline of engineers from China and India dries up.
“It’s a subject I was among the first to write about a decade ago…”
.
(1985) Nagle. ‘On Packet Switches with Infinite Storage.’ IETF RFC 970.
(1999) Jacobson et al. ‘R.E.D. in a Different Light.’ Cisco Systems.
(2009) Reed. ‘[E2E] What’s wrong with this picture?’ End2End-Interest.
.
And, though not identically themed, van Beijnum, via ArsTechnica, ~72 hrs after Cringely’s Jan 2011 blurb:
.
“Cringely got wind of the problem through the blog of Bell Labs’ Jim Gettys… Cringely gets many of the details wrong.”
.
I do think there is technical merit to this writeup, and Cringe’s advice seems to be legitimately helping a few people; those things shouldn’t be undervalued. The ceaseless name-dropping, self-promotion, and regurgitation of others’ earlier work serve him less well.
This “prediction” is just a load of attention grabbing spin. It’s like a UK tabloid newspaper hack writing about trains delayed because of one inch of snow. (25.4mm for pedants.)
.
I expect Mark Stephens (aka Cringely) is hoping the serious media pick up on this and he will be in with a shout again. Imagine – the big studio. The lighting. Pompous music and spinnign logos Zooming in camera. Serious look to camera and then a hard glance and panning camera to Mark Stephens Yes! Yes! The moment has come!
.
Then he woke up. The twenty year old suit was still hanging in the cupboard. Receding hairline with a whiff of Grecian 2000 hanging limply across his face. Some stray hairs gently flapping in and out and in and out to the inhalation and exhaust of his nostrils. Still, yah know, on with the threadbare robe for a pad across the carpet and through the door to the large cupboard used as an office in a now reduced circumstances house. Fingers cracking less with go get ’em energy and more arthritis (but it’s the though which counts) wave above the keyboard like a conductor preparing. Prediction five begins!
.
On the last full stop (period for colonials) a long deep sigh. Ah yes. Punch the air in slow motion like he saw a CEO whose name nobody remembers and cannot pronounce of, what was that company? Micronerf? Macroversion? Mickysoft? Giggle? Goggly Goggly. No, not that. Goo… Goo… No it will come later.
.
“Ooh, ooh. Press publish. You know you can do it you big sex pot you.” floated through his mind. Will they love it? Will they hate me? Five hours later biting his knuckles and staring wide at the wall in a curled up ball. “No. No. Tomorrow. Tomorrow. It’s all in Flux man. It’s all in fluuuuuuuuux.”
.
Sorry. Bored as you can tell.
Trashie!
Are you sure you haven’t inadvertently connected to my home camera? That certainly sounds like a description of me.
You naughty girl! Cheeky monkey!
My suit is actually closer to 26 years old. Its the one I got married in. It no longer fits. I did buy a larger jacket and trousers along with a new shirt & tie for my mother’s funeral. Wore it again six months later for my sister.
How’s it going?
Life, for some; death, for others; and nothing for me thank you!
Buffer bloat is real, but it exists in edge software boxes where it’s cheap to do so. On backbone hardware boxes queueing is expensive and market drives for as little queueing as possible.
Modern devices such as CSCO SIlicon1 or BRCM Jericho2 do not even have bandwidth that traffic CAN hit delay buffers. Existing devices like MX or ASR9k are fully buffered, all traffic hits delay buffer, more dense modern devices have like 20% of delay buffer bandwidth, so even in theory only 20% of traffic could hit delay buffer, rest just cannot experience even a microsecond of delay.
In a large tier1 transit network, which runs older fully buffered devices, in any given day across globe average 1way jitter is 2.3us and max jitter 15us, these numbers are several orders of magnitude smaller than anyone who doesn’t measure this thinks they are.
These are not low because the networks are carefully designed for that, there is no customer demand, they could be 1000 times more, 2.3ms and 15ms and customers would accept that. It is purely emergent feature of economic design of a dense fast device.
Re: “I have an Eero mesh network with five routers in all… But here is something important: one reason my Eero system is good is because my routers are all older units.”
I thought a home network, mesh or not, has one router and several access points. I can’t imagine 5 routers, unless he’s talking about 5 subnetworks or separate networks. Also, the whole point of a mesh network is to have all the access points talking to the router, passing along its data, as a single system, whose components work together, implying they were designed together, by Eero in this case, and are therefore the same age. If the “routers” are “older”, how could they all be Eero?
I was reading up on this topic — innocently — to bolster my own grasp of 802.11 mesh gear.
.
I stumbled across the following, dated November 2020 and December 2020, respectively.
.
https://www.cnet.com/news/eero-6-review-amazons-wi-fi-6-mesh-router-underwhelms/
https://www.theverge.com/22193059/eero-pro-6-review-mesh-router-wifi-6-test-speeds-network
.
Give those a gander. Do their subject matter or phrasing look at all familiar?
@Questionable_PhD
I’m not spotting anything here. CNet had a more punchy tilt with a “Yes, but…” than the Verge which seemed on a downer and backward looking from the beginning. My eyes glaze over at most tech reviews as many are paid for by the yard. I don’t have much time for most makeup reviews either. I don’t want to hear half an hour of someone’s life history with five minutes worth of content sprinkled throughout. I’m about as interested in the latest “mesh gear” having no use for it as you guys are interested in the latest SP25 foundations.
.
Mark Stephens (aka Cringely) doesn’t get systems or how systems interact hence his old man desk banging. “Bufferbloat” is a marketing word he can pull out like a stick and beat the air. Really the whole “prediction” is has “Swampmonster 2: Return of the Swampmonster” reduced budget straight to DVD cash-in written all over it. This from the “journalist” who claims not to do “second day stories” while his last “prediction” was a reaction to current news events at best.
But at least bufferbloat did return, thanks to ISP greed.
Mineserver did not turn up, thanks to Cringely greed.
What do you think Mark Stephens, will it ever turn up?
Two more of Mark Stephens (aka Cringely) “predictions” to go. I’m curious what self-promoting glib yarn he’s going to tell about Mineserver and which variant of emotional blackmail it’s going to be this time.
.
Perhaps some dreary story about his people calling their people, how Steve Jobs had a personal message for him he never forgott, Bill Gates phoning his poker pal Warrent Buffet to get in on the deal but it all fell through because Buffet had finalised a maga-deal and had no immediate cash on hand, Vlad who he made contact with during his post-Soviet flirt with being a “financial journalist” is good to go for bringing a few oligarchs on board to provide fifty percent finance if the Nigerian prince is on board too, but he’s going for the NASDAQ option so needs every else to sell their houses and tap their wives for finance so Facebook will buy him out for fifty billion. Oh yeah and his bestest ever mate George in Portsmouth who he emailed with twice says it’s the real deal so that told all the naysayers “Nee ner”.
.
Juts kidding. Mark Stephens is backing a new start up using recycled USB stick microcontrollers to give you a supercomputer in a pocket with 1000 teeny tiny CPUs in parallel giving a 10,000x performance increase for a fraction of the batter power and magic handwavy memory so you don’t suffer bufferbloat and drive storage provided by thin coated clingfilm folded ten (!) times which can also double as a 99.7% efficient solar energy collector.
There are/were thousands of people involved in fixing bufferbloat for the world. Sure jim (and I) kicked it off, but many key contributions were made by names known and unknown.
I didn’t actually regard the work as “done” til 2015 or so, but it is in general my hope that a lot of consumer routers do actually have most of the right things in them, they just need to be turned on and configured properly by the user or isp.
How everything works was written up
in the free book: https://bufferbloat-and-beyond.net/
Don’t forget as well, the more upload speed you have on cable modems, the chances are you’re using more channels and because of the way the cable modems work, that means there is additional delay for each additional channel used thanks to TDMA.
The cable modem has to wait for a free slot to send and receive, the more upstream channels you have, the more it has to wait during the cycle, it gets worse as well as cable modems are a shared resource, with contention shared among any one on your local UBR.
And because you need aroung 10% overhead for TCP in upload vs download speeds, dropping your download speed will also (usually) drop your upload speed, and giving you less latency.
Faster is not always faster, more so when it comes to streaming content and definitely not when it comes to online activities that you don’t want jitter or latency on, such as gaming.
I know that when AT&T GigiPower delivered 1000 MBytes up and down in Cupertino, the only router that could handle is that was less than $2,000 was a $60 Ubiquity unit. Not sure if it was bufferbloat features or just quality engineering, they still run my home office network.
The mandate to offer base broadband across the US should include consumer protections against ‘bandwidth’ faux-engineering to up-sell pointless products. I really wish there was more competition in the market, and indeed municipalities were encouraged to push out a lot of the junk bandwidth the for profit companies ‘say’ they are offering to less to do neighborhoods.
P.S. For a safe and productive Internet, please don’t feed the trolls.
amazing content !
This is really a extremely beneficial read for me, Have to admit you might be 1 in the most effective bloggers I ever saw.Thanks for posting this informative article.
Thanks for this useful article.
WiFi 6 is a bust, but WiFi 6E is a totally different animal. It takes the protocol improvements from WiFi 6 and places them on new, uncongested spectrum (and there’s a lot more of it). This means that areas with dense 5GHz utilization should see an improvement in reception. This will help significantly once equipment is available, which mostly won’t be for 6-12 months.
Don’t forget about all those cars Musk keeps churning out. They’ll eventually use Starlink.
What a great and interesting post. I like your writing skills and appreciate your working effort making on this post. Street View Online