There’s a peering crisis apparently happening right now among American Internet Service Providers (ISPs) and backbone providers according to a blog post this week from backbone company Level3 that I am sure many of you have read. The gist of it is that six major ISPs of the 51 that peer with Level3 have maxed-out their interconnections and are refusing to do the hardware upgrades required to support the current level of traffic. The result is that packets are being dropped, porn videos are stuttering, and customers are being ill-served. I know exactly what’s going on here and also how to fix it, pronto.
The problem is real and Level3’s explanation is pretty much on target. It’s about money and American business, because this is a peculiarly American problem. Five of the six unnamed ISPs are American and — given that Level3 also said they are the ones that typically get the lowest scores for customer service (no surprise there, eh?) we can guess at least some of the names. According to the American Customer Satisfaction Index’s 2013 report (the latest available with a new one due any day now) the worst ISPs in American are — from worst to less bad but still lousy — Comcast, Time Warner Cable, CenturyLink, Charter Communications, AT&T U-verse, Cox Communications, and Verizon FiOS. That’s seven companies and since Level3 says only five are creating this peering problem then two in there are off-the-hook but still not the best at what they do.
The idea here is pretty clear: these five ISPs want to be paid extra for doing the job they are already being paid for. Extra ports are required to handle the current level of traffic and these companies are assuming that when the pain becomes great enough — that’s our pain, by the way — Level3 or some Level3 customer like Netflix will pay the extra money to make the problem go away.
This ties into the current Net Neutrality debate and the new FCC rules that Chairman Tom Wheeler says he’ll be offering-up later this month that will both keep the playing field level while somehow allowing for a version of fast lane service. I already have doubts about Chairman Wheeler’s proposed rules.
Let’s understand something: Internet service is an extremely profitable business for the companies that provide it. Most on this notorious list are cable TV companies and generally they break even on TV and make their profit on the Internet because it costs so little to provide once the basic cable plant is built. So what these five are saying, if Level3 is on the level, is that the huge profits they are already making on Internet service just aren’t quite huge enough.
I’d call this greedy except that Gordon Gecko taught us that greed is good, remember, so it must be something other than greedy.
It’s insulting.
So here is my solution to the problem. I suggest we look back to the origin of peering, which took place in the dim recesses of Internet history circa 1987. Back then the Internet was owned and run by the National Science Foundation and was called NSFnet. Lots of backbone providers served NSFnet and also built parallel private backbones that were generally built from T1 (now called DS1) connections running at 1.5 megabits-per-second. Most backbone links today are 10 gigabits-per-second and there are often many running in parallel to handle the traffic. Back in the NSFnet days peering was done at a dozen or so designated phone company exchange points in places like Palo Alto and San Diego where backbone companies would string extra Ethernet cables around the data centers connecting one backbone with another. That’s what peering meant — 10 meters or less of cable linking one rack to another. Peering was cheap to do.
Peering also made for shorter routes with fewer hops and a generally lower load for both backbones involved, so it saved money. Nobody paid anybody for the service because it was assumed to be symmetrical: as many bits were going in one direction as in the other so any transaction fees would be a wash. Most peering remains free today with Level3 claiming that only three of its 51 peers are paying (or are being paid, it isn’t clear which from the post).
The offending ISPs are leaning on the idea that with Content Distribution Networks for video from Netflix, YouTube, Amazon, and Hulu, the traffic is no longer symmetrical. They claim to be getting more bits than they are giving and that, they say, is wrong.
Except it’s actually right (not wrong) because those bits are only coming because customers of the ISPs — you and me, the folks who have already paid for every one of those bits — are the ones who want them. The bits aren’t aren’t being forced on the ISPs by Netflix or Level3, they are being demanded from Netflix and Level3 by we, the paying customers of these ISPs.
The solution to this problem is simple: peering at the original NSFnet exchange points should be forever free and if one participant starts to consistently clip data and doesn’t do anything about it, they should be thrown out of the exchange point.
Understand that where there were maybe a dozen exchange points 25 years ago, there are thousands today, but if a major ISP or backbone provider doesn’t have a presence at the big old exchange points — that original dozen — well they simply can’t claim to be in the Internet business.
These companies are attempting to extort more millions from us just to provide the service we have already paid for.
I say throw the bums out.
Thoughts triggered by this post:
1. I haven’t read Michael Lewis’s new book Flash Boys but your description of peering sounds an awful lot like what Lewis has discussed in interviews about how the high-frequency traders created their speed advantages. Fastest connections win.
2. I have read Jaron Lanier’s Who Owns the Future? His contention is that the the biggest computers win.
3. The Net Neutrality debate seems to be framed as the companies you mentioned vs. their customers in terms of who has the greater influence on the law/policy-makers. “Greater influence,” of course, means the most influential money wins.
Not to be overly pessimistic, but I can envision a perfect storm of sub-optimization whereby the companies you name strangle their customers and the American Internet economy in their myopic quest to further their own (and their shareholders’, of course) “best interests.”
I think everyone would be more open to hearing Comcast’s side of the story if they weren’t on a shopping spree. Comcast has been and continues to upgrade their network -they now have as many as 16 6MHz channels per node just for Internet traffic and 40 GigE rings connecting hub sites- but instead of telling that story and focusing on improving what they have, they decide to go after Time Warner. Maybe buying a few isolated TW sites to get better regional nets in place makes sense, but the whole company? No way is that going to go over well with anyone.
In the UK I’ve heard that some internet providers already charge and receive monies from certain companies that provide content. They weren’t happy with the BBC started iPlayer because it was instantly popular and much was downloaded, in other words doing exactly what the internet provider advertise that you can do with their service!
.
So, although I don’t quite understand how in the UK they’re able to charge, I do completely agree with Bob that internet providers must provide content demanded by the user. If they aren’t able to provide then they should be handing back the monies charged to the customer.
.
Once you start putting in place a mechanism that ISPs must hand back some, or even all, of their charges for failed service levels you’d be surprised how quickly they’ll improve.
Remember, LINX is not-for-profit, and is for the benefit of all the ISPs in the UK (and others who touch the UK). As I understand it, the main LINX peering points are literally nothing more than big fast switches. If you want more, you add more. When iPlayer was really getting big, the ISPs started falling over themselves to peer directly with the BBC, to reduce their transit costs. I think you’ll find most the major UK ISPs and networks (including the likes of Janet, who runs the academic network) are peered now, and a traceroute from one ISP to another shows a pretty direct route, without hitting a tier-1 provider like Level 3.
There’s two ways you can play it – either it’s mutually beneficial, so we all agree to connect to this thing to save us all money (i.e. LINX) or you do your own thing and meet via a third party (a la US ISPs). I understand there is some peering, and the geography of the USA certainly doesn’t help with peering, but I can’t help but feel it’s mostly the fact the US ISPs simply don’t want to get along. Given all the recent noise over net neutrality and the likes, it seems it can only get worse.
I thought ISPs putting download caps on here in the UK were bad, but I think the USA has much worse coming, unless the FCC gets a clue.
Umm.. yes Linux is for profit. I think you’re confusing what open source is and what non-profit.
LINX is not Linux. You’re in such a hurry to correct someone you don’t even realize you have no idea what he’s talking about.
No one said linux you idiot, don’t talk about computers if you don’t know what you’re talking about, people like you are why net neutrality is becoming a dinosaur….
London InterNet eXchange .. am the hundredth commenter ? do i win a prize ?
Also, it so happens that Linux (Not LINX) actually IS Not-for-profit. From Linux.com. “The Linux Foundation is a non-profit consortium dedicated to the growth of Linux.”
BTW: I just finished reading “Where Wizards Stay Up Late: The Origins Of The Internet.” on audio. That NSFnet graphic really puts things in perspective.
[…] via I, Cringely Level3 is without peer, now what to do? – I, Cringely. […]
Hi Bob,
Obviously your target market is the USA, but do you have any information on this type of problem in Canada, or the rest of the world for that matter? Since it is the WORLD wide web, just wondering if the same problems are rampant everywhere. I know for myself there are only two real ISP’s (Rogers and Bell), so for competition, probably not much better than what the US has.
Thanks!
Joel
Given that cable ISPs have a monopoly (or close to it) on local service, I suspect Canada’s ISPs are much better.
Time Warner Cable has been at the bottom of customer satisfaction since the 1980s. Satellite networks such as DirecTV have made that the centerpiece of their media campaigns, but they’ve recently gotten into trouble by changing their signup terms to 24 months of commitment instead of 12.
My understanding is that it’s a (semi) uniquely US problem. Countries whose ISPs charge per amount consumed (Japan, S. Korea) don’t end up with the same problem, they also end up with much higher quality of service. The US system doesn’t directly incentivise ISPs to perform.
Jeff, I disagree with the idea that unmetered connections are cause issues, many US ISPs do meter their connections, and it doesn’t improve quality of service. In addition, in the Netherlands you can get unmetered 70mbit connections for cheaper than DSL in the states.
As a resident of South Korea, we do not have metered internet. Korea’s ISPs are pretty much same as US ones. But cheaper and faster. 100Mbps is around $30/mo. You pay and consume data as much as you want.
How do we throw them out? I have two choices of internet provider, Comcast and Verizon. One is over priced the other is a horrible service. I have no other choice in providers because, as I understand it, right of way laws prevent competitors from laying cable and offering service.
I would love to throw them out, but how?
Throw them out of the exchange point, not out of business or out of the Internet. There are thousands of exchange points presently. Every data center with more than one backbone provider is a potential exchange point. And most of the peering traffic doesn’t go through the original 12. But they retain symbolic importance. The entire point of Level3 going public on this problem was to embarrass the ISPs into doing the right thing. As the US telecom regulator, the FCC could easily impose the rule I have suggested. If you want to peer then you have to be equipped to handle the traffic. This is Restraint of Trade, folks — a violation of the Federal Trade Act — made worse by the likelihood that it’s being done in collusion. Where is the Department of Justice?
Where is the USA Justice Department? They’re busy looking the other way. Just as they’ve been doing with IBM for the last 10+ years.
in fact, companies used to be blocked at peering points periodically, back in the days when The Connected Internet was coming to grips with the new usage rules that allowed commercial traffic, not just academic use. not often, but it did happen, if CrankCo wasn’t being nice to the user community. seldom lasted more than a few hours. nowadays, it would start a riot.
I could whine about asymmetrical traffic and all the rest like some of the big boys do at the top of the office tower, but the answer is quite clear. go big or go away. if you can’t deliver, somebody else will, and hike the price in six months because they can.
tech is like that. every 18 months, your stuff is laughable. that’s why Wall Street casts a fishy eye at tech, there’s all this damn Research and Reinvestment and Upgrading in the way of pulling a rip on pocketbooks and passing it down in dividends.
> Where is the department of justice?
Waiting for someone to report this violation?
Short of implementing effective anti trust laws directed towards the ISP’s. The more viable alternative in my opinion would be a organized coalition of grass roots movements. Encouraging local governments to build their own back bones. Offering their citizens a competitive alternative to the major ISP’s doing business in their communities. The question is, can a grass roots movement grow to be strong enough to thwart the big money influences which created this mess in the first place? The good news it has been done elsewhere in this country.
They can’t. Existing ISPs in most areas have paid off cities/counties/states to not allow local infringement into their business.
It’s not necessarily a “pay off”. The idea of “natural monopoly” has been around a long time. Like water, electricity, phone wiring, and cable TV. And let’s not forget about busses, trains, and taxi service. Certain monopolies do seem natural and in the public interest. In order to insure the public interest is maintained, they are allowed to exist as “regulated monopolies”. Unfortunately, when non-technical people do the regulating, all they can think about is how to keep costs down, which stifles progress. That’s why the water and electric companies have to advertise to tell us how to use less of their product.
Kaboodle might be referring to the twenty states that have banned or highly-restricted municipal ISP’s – http://arstechnica.com/tech-policy/2014/02/isp-lobby-has-already-won-limits-on-public-broadband-in-20-states/.
I can remember reading years ago about the future of the internet that the cable internet providers wanted and it is just like their tv subscription model. Tier 1 gets you “basic” internet package with say 10 websites. Pay more and you get tier 2 which gives you access to 20 or more websites. Pay even more and you get 50 or more websites. Oh, and by the way, you don’t get to pick which of the websites that you are paying for. It’s just like their cable subscription model.
All I could find was this one: http://i.huffpost.com/gen/1567010/original.jpg . This one is updated from the one I remember (the same one I think you are referencing) but conveys the same idea.
I was right !
It IS a bunch of tubes !!!!
(just kidding, Not the real guy, but someone should mention him/me)
RIP Uncle Ted. He was pretty good for Alaska, and had one of the most apt analogies for the internet, despite it’s humor value.
Why do I get 60 Mb/s down, but only 4 Mb/s up? I am paying Yankee dollars for this.
This is like those childhood games to see who can endure the most pain. I say we not pay the shakedown money, let the system degrade, take the pain. At some point competitors should emerge, and we should pressure on all legislators and regulators to do everything to open the way for competition.
And Level 3, please name names. Some naming and shaming is in order.
This is the real problem. There is no competition. If the ISP’s would allow local cities, municipalities and governments to lay their own pipe without the threat of lawsuits then this issue would be moot. Let them all buy each other up if we admit that the internet is a utility just like water, power, gas and sewage; and it’s regulated that way.
The head of the FCC is going to lay down these rules then take a job with Comcast or whoever offers the two million dollar plus a year contract.
The net neutrality guys are missing the issue. We need to promote competition.
I live in NYC and the internet here is like living in a third world country. I won’t have Time Warner because it will go out for a week at a time for $50.00 a month so I’m stuck with Verizon DSL because Verizon FiOS has failed to connected to enough buildings as they were contractually obligated to do via their franchise agreement with the city. So I’m stuck paying for $50.00 a month 2mbit DSL AND $50plus a month on a verizon and t-mobile phone when I have to tether to make up for the poor DSL.
I wish NYC would cancel their franchise agreements and/or go into competition with the local ISPs.
it’s pricey to overlay another network, very pricey to duplicate everything.
I have long opined that one core network, pricing and distributing by way of SNAP packets or other headers, is the way to go.
but that’s too disruptive for the economic system we have.
This is exactly why Eastern European cities have great Internet connections today. There was grassroot growth, started by local neighborhood mini-ISPs as early as the ’90s, who dealt with issues such as peering ruthlessly, in exactly the manner you described: put up or shut up. Two decades later, the fierce competition means that everybody plays nice and does their best to satisfy their customers, or else (they bail).
Incidentally, the exact same pattern emerged for telecom and mobile operators, so now some Balkan countries have 60% or more of the population on prepaid plans, which means paying month-to-month or even not at all (pay nothing and still receive calls and texts for a few months grace period), no ties whatsoever, and even the ability to take your phone number to the competition at any time (customer owns the number, once allocated). Basically customer can bolt at any given moment, which unsuprisingly keeps operators at their best behavior.
Who would have thought that competition can be great for the customer.
While you are right about Eastern Europe, the comparison to the USA is bit off. In the early 1990s, the USA had lots of small competitive / cooperative ISPs, too. I know; I ran one in a major metro area. But we always knew that the big stupid phone companies and cable companies would eventually get into the game, and do their best to squash us. At the time, my company and half a dozen other small startups were the only real game in town for Internet access. We competed for customers, and cooperated in educating each other and obtaining better backbone connections. PhoneCo and CableCo didn’t have a clue.
20 years later, all but a couple of those small ISPs are gone, and mostly have specialized into niche markets, hosting, etc. because PhoneCo and CableCo paid lobbyists to pass laws and regulations that favored them, cutting the small ISPs off at the knees as to speed and bandwidth. Of course, we always knew the big guys would have a larger captive market to sell to, and could probably sell at a lower cost — but we had hundreds or thousands of loyal customers who preferred our far better customer service and network reliability.
But ultimately, when all a small ISP can sell is 1.5MB DSL and PhoneCo can sell 7-20MB DSL, the customers are going to leave.
Re: “But ultimately, when all a small ISP can sell is 1.5MB DSL and PhoneCo can sell 7-20MB DSL, the customers are going to leave.” True. But we have a form of DSL and higher speed competition now, Like DSL Extreme and Covad/Megapath. But they use Telco installed wires and buildings. Are you suggesting the competing ISPs should be allowed to dig up the streets and build their own buildings to house the switches?
Relevant: https://www.youtube.com/watch?v=NAxMyTwmu_M
Haha that’s what got me reading up on this issue. There’s also a blog format of the same thing (with the video) here: http://vihart.com/net-neutrality-in-the-us-now-what/
.
Anyways here is Netflix’ latest post: http://blog.netflix.com/2014/04/the-case-against-isp-tolls.html
Level 3’s latest: http://blog.level3.com/global-connectivity/observations-internet-middleman/
And Mozilla now getting involved: https://blog.mozilla.org/netpolicy/2014/05/05/protecting-net-neutrality-and-the-open-internet/
Many years ago I acted on one of Mr. Cringely’s ideas and built a “ClarkConnect” server for my home. One of the things I like about it is it provides reports on my Internet usage. Here are a few revelations — (1) I get virtually no content from my ISP, zero, nada. ALL of my content comes from the outside and passes through them. (2) Most of my bandwidth is used by advertising, over 80%. We use HULU and Netflix; and they are not even close to the advertising services. Why don’t the ISP’s squeeze the advertisers? I’m not pay for them. Actually I am, but I don’t want to.
.
I was once a VoIP customer. For a few years the quality of the service was acceptable. Not perfect, but good enough. Then the service went to crap and no one could figure out why. At the time I was running a web hosting facility and decided to conduct an experiment. I set up a test VoIP service. I used the same port numbers and technology as did my VoIP provider. I ran some tests and confirmed my suspicions. My ISP was restricting the bandwidth used by VoIP. They wanted you to buy THEIR service, so they made everyone else’s run poorly.
.
I then realized interfering with phone calls is a dangerous thing. On further thought I realized my mom medical alert system uses VoIP. Her home security system uses VoIP. By messing with the VoIP bandwidth my ISP was potentially interfering with emergency and life saving services.
.
According to my ISP, I am getting over 30 mbs of service. On closer examination I found out that is a compressed data rate. The real rate is about 15 mbs and that is a peak rate. The rate I can expect most of the time is about 6 mbs.
.
I’d like the FCC to enact the following rules:
.
1) Require ISP’s to sell their bandwidth based on real rates that they can guarantee 99.99% of the time. No more peak rates, no more tricks. And that rate is with data sources OUTSIDE their network. Yes they can over subscribe their networks, but they have to be honest about what you can actually expect to get to your home.
.
2) It is okay for an ISP to limit bandwidth to one’s home or mobile device — but it must be done with full disclosure. That limit should be well above the 99.99% rate you are paying for.
.
3) Require ISP’s to protect bandwidth for critical services. That would include VoIP.
.
4) When ISP’s need to prioritize bandwidth, the lower priority should be assigned to services that are not time critical. Do you care if your file transfer takes 20% longer? No. Do you care if your video stream gets messed up? Yes.
.
Life critical applications must be given TOP priority. Time critical applications must be given high priority. Non-time critical applications get the lower priority.
.
5) ISP’s should be required to all streaming content providers (like Hulu and Netflix) to operate staging or cache servers on the ISP’s network. This would improve the quality of service to the ISP’s customers AND reduce the bandwidth used to the Internet backbone — saving the ISP money.
.
6) Prime time and not — If I am paying for a solid (99.99%) 5 mbps service, that is during prime time. 99.99% means that is the minimum service one can get at ANY time. If you want to give me 60 mbps service outside of prime time, great.
.
When these rules are applied to mobile devices, we will quickly realize most Telco’s will not be able to provide large data rates 99.99% of the time. There are physical limitations on what can be done with their networks. This is where the above rules are even more important.
.
I live in an area where their is periodically severe weather. We get weather alerts over our phones. That is a very good thing. However when we’re sitting at a ballgame the cell phone network can’t handle the load. This is a situation where prioritization must be done better. Weather alerts involve public safety. They must be given priority over most other forms of network usage. Staging/caching information by cell tower would be a very good thing. Broadcast alerts to all subscribers in range of a given tower would be a good thing. It is very frustrating to see a nasty storm heading for you and your phone’s network isn’t working. Do you need to find shelter? How much time do you have to find shelter?
.
On the general subject of the Internet, improvements, neutrality, fraud, communications, etc:
.
Verifiable user identity. On the Internet I should be able to know exactly who I am communicating with, and them me. When we can verify identity, we can then end spam and thwart many other forms of Internet problems. I would make this feature “optional.” If people wish to communicate anonymously, they can. I want the option to block anonymous communications with me and my family.
.
Secure and reversible financial transactions. Lets face it, there is a lot of cyber crime these days. When fraudulent transactions are made, they should be fully reversible. The consumer or his/her bank should not bear the financial loss. The banks involved in the transaction should be required to return the money. If a bank somewhere doesn’t want to cooperate, then they don’t need to be part of the worldwide banking system. Everyone must play by the rules, or they are out.
.
It is time to eliminate most telecommunications tariffs. All phone calls in the USA should be included in the basic service and at a fix price. The concept of long distance or non-local calls should be eliminated. There should be no need for toll free 800 numbers. The surcharges for calls between most countries should be eliminated.
.
Verifiable identity and everyone playing by the rules applies to VoIP too. I am fed up with all the phone based scams, efforts to trick my aging mother, etc. I expect everyone who calls my home to follow the laws and rules. If you don’t you should be blocked from the USA phone system. All business calls must have valid caller-id’s. What possible reason would a legitimate business want to hide their identity? All calls must be traceable to the originator. All phone (and VoIP) providers must know who their customers are, their real identity, their address/location, how they’re paying for their service, etc.
.
The Internet has opened up communications worldwide. This has transformed business worldwide. It is now time for the other forms of communication to open up and this will benefit worldwide commerce too. But we must also follow basic rules to insure honesty and integrity, and protect people against fraud.
Oh, I’d love this. Particularly any support around SPAM calls. I am tired, but still somewhat amused, but the support team for “Windows Company” calling to tell me either I have been infected with a virus or that my windows update isn’t working. I love trying to find out who they work for, “Windows Company” is not a legal company, and why they think my Windows box is the one infected. The last time, a few days ago, they called after my Windows box had been off the internet for over 2 weeks.
As I said, tired but still enjoying talking circles around these scammers.
When the banks of these firms start having to give back the money, I suspect the telemarketing callers will quickly go out of business. When you can trace the money to a real person, then law enforcement can do its job. Since many of the operators are offshore, it may not be possible to convict them. But if legal system then has the ability to block their access to the global networks — that will provide enough incentive for the local governments to clean things up.
I had the same thing happen to me about four months ago, right after I spent an evening running some maintenance on the computers in the house.
I was amused that the guy had the stones to stick to his story when it was plain that I didn’t believe him.
Next time, I’m gonna see how fast I can get them to hang up by saying we’re actually with the FBI, we’ve been tracking them for weeks, and we currently are running a trace on the phone call. We’re aware this is a scam and that in 20 minutes, we’ll be at your door, so if you try destroying any evidence it’ll add onto your prison sentence that we’ve already determined, which is a minimum of 14 years.
So this company was doing you a service, helping you out after your computer was down for 2 weeks.
You know, I never thought of it that way. I may just have to write the “Windows Company” a Thank You note!
John, you are a dream come true for congressmen, big-company lobbyists and the NSA. You are asking for exactly the opposite of net-neutrality. Sadly, you will probably get your wish, reminding me of HL Mencken: “Democracy is the theory that the common people know what they want, and deserve to get it good and hard.”
John said “Most of my bandwidth is used by advertising, over 80%”. It’s true that there are thousands of ad servers providing ads trying to tag along with our content. But they can easily be blocked (i.e. never downloaded) with a readily available hosts file (or other ad blocking software) together with an email service provider’s blacklist for email spam.
When a port is overloaded and the peer will not upgrade it, why doesn’t Level 3 just drop them already? In fact, when a peer demands Level 3 pay them, why doesn’t Level 3 just fire back asking to be paid themselves or they’ll drop the peering immediately. They can always back down to a free arrangement again once it’s fixed.
It would seem to me, that an ISP that doesn’t have any connections to any backbone providers isn’t an ISP at all, as Bob mentions. And, while people may not be able to switch, they can sue that they’re not getting what they paid for, and that the service is being falsely advertised when it cannot connect to the Internet, but only some subset. Seems like an easy class action win.
Right now, it’s a harder case, since service is being provided, it’s just poor.
When the broadcast networks and the cable providers fight about transit fees, that’s how they wage war by removing the content and running ads. I would think this would be crazy easy for the backbone providers to do. Imagine this ad from Level 3: Comcast has been dropped as a peering partner because they refused to pay $30,000 to upgrade their equipment that connects with us for free. Instead they demanded we pay them millions of dollars so the customers that already pay Comcast for access to the Internet can actually access the Internet.
Drop one peering location and force the traffic to travel farther to another one, and probably overload that one then and repeat. How many days of of severely limited Internet would it take before this is fixed. I would bet that it’s significantly faster than the week of TV outage that happens sometimes.
That kind of catastrophe is probably what’s needed. Make it worse to clearly expose how ridiculous it is and get it fixed.
Comcasts interfaces are on par or faster than L3’s… so no.
So where is the bottleneck, exactly?
While Bob’s idea about “the original dozen” peering points is an interesting one, I immediately thought of the “Peering routes” table that Level3 included in Monday’s post. While in 1988-1989 Palo Alto (for example) was Northern California’s only peering point, it appears from the Level3 table that today it is small potatoes — San Jose now has three peering points to Palo Alto’s one. Ann Arbor, Boulder, Champaign-Urbana, Ithica, Lincoln, Pittsburgh, Princeton, Salt Lake City and San Diego are all on the 1988-1989 map but none of them are shown in the Level3 table. Today Level3 has three peering offices in Chicago and three in Los Angeles; neither of those cities show up in the earlier map, and neither do New York City (2 points) nor Denver, Miami, Tampa, nor Tustin. The original peering points were based on the hometowns of the universities that were the initial customers of the NSFNet, but it looks like today the major peering points are the nexus of where the largest number of customers live.
way back when men were real men, and real computers had tape drives and consoles eating reams of paper, there were local peering points between all the ISPs and major customers. local peering points were a co-op, at or near a university computing center. that’s because the universities were the gateways to The Connected Internet.
next stage was mostly peers operated by the major computing companies. Palo Alto and Boston were DEC centers, for instance.
today, the minor peering points are in telco colocation centers and the occasional “data hotel,” and many of the national centers are telco points or nexus like a Googleplex, where everybody drops a 10-gig or two, and the backbone router may allow comingling of the data streams.
all peers are congested to one point or another. play with traceroute sometime. any time the delay rises, the route is congested. a 60 mS traverse used to be the outlier back when I still had some hair.
“…to provide the service we have already paid for.” We paid for infinitely expanding peering? And people say the government doesn’t set up sustainable services! Awesome!
This is silly. Sure, there’s quite reasonably arguably some form of price-gouging going on here. But the “pipes” and their maintenance are not free, and usage has exploded and is continuing to grow.
When I pay my local utility for water service, I fully expect that tap to run full blast on demand, whatever time of day I want to use water.
As the town grows it is the resposibility of the water company to upgrade their infrastructure as new subdivisions are built and new customers sign on and also demand more water delivery. It’s a perfectly normal cost of doing business.
This is how business normally works. As you grow your customer base, so must you grow the infrastructure that serves them.
The difference is we didn’t pay for all those drops of water already, each and every drop gets charged to us. So if there’s a Netflix of water usage and your water usage jumps 10 times, you can definitely expect your water bill to also jump 10 times.
Sounds a bit optimistic. The water system is a regulated monopoly, so before they upgrade anything, they declare a “water emergency”, pass water conservation laws, including restrictions on use (when and for what purpose). Spending money on upgrades is usually the last thing voters and politics will allow out of fear of being ripped off by the utility. Same goes for the electrical system.
I used to run a web hosting service. At one point we had 6 Internet providers with demarc’s in our data center. I will say the level of service and professionalism of Level 3 was exemplary. I have a lot of respect for that firm.
.
Figuring out how and when to use 6 Internet providers was an interesting problem. Sometimes a customer would insist we put their websites on ABC’s network — okay fine. Some could handle peak traffic better than others. Some would charge us less for peak traffic than others. Some had too many hops to the end users. Some really wanted our business…
.
Every day I would drive to and from work on a very nice Interstate. The USA has a very nice highway system. The better systems are based on good standards and design rules. It transformed the USA economy after WW2. After tough days at work while driving home I’d wonder — why don’t we have a standard national telecommunications system and infrastructure? Would we be better off if we combined all the intercity and interstate voice and data networks, operated them as one system, set up a non-profit corporation to manage them, etc? The highway system was built and is maintained by transportation taxes. Why couldn’t our telecommunications system operate in a similar manner?
.
I had to deal with the telco’s crap as part of my job. Now they’re pulling their crap in our homes. Maybe it is time to take the work away from them.
You can thank the breakup of Ma Bell for that. Of course, when the Bell System broke up, there was a burst of competition and activity, but that has since gone the way of the dodo.
In large part because the parts of Ma Bell were allowed to buy each other up until the remainders are in fact bigger than Ma Bell ever was in the first place.
“The highway system was built and is maintained by transportation taxes. Why couldn’t our telecommunications system operate in a similar manner?”
This goes back to the creation of the Internet. The original design utilized the existing infrastructure put in place by the telephone company. It was already in place and duplicating a network that size would have been cost prohibitive.
To put things in perspective, Level3 boasts on its website about spending $40 Billion on its network. If you look at AT&T’s numbers, their network uses 8 times as much wiring.
Level3’s network is equivalent to the U.S. Highway system. AT&T’s is the equivalent to all the roads and highways not part of that highway system. And it only represents a portion of the countries non-highway roads. (Comcast, Verizon, etc. being the other road systems.)
The amount of money those companies have spent connecting consumers to the “highway” runs into the Trillions at this point. There is no way the Federal Government could allocate that kind of money to building a network to connect consumers to the Internet, not with a 17 Trillion debt. (Especially when the network already exists.)
BTW, you might want to check on the status of the highway fund. The Congress has basically misappropriated the funds meant to build and maintain the highway system. The money that should be spent repairing roads and bridges is instead spent on Congress’s pet projects.
Duplicating the highway fund with an “Internet Fund” just gives elected officials more money to redirect towards pet projects.
And, of course, since the U.S. is not spending nearly enough on its highway system and other infrastructure, they are crumbling. In ten years people will probably be saying “How come our highways and bridges aren’t as good as our internet?”
There are two partitions that are currently active on the whitehouse.gov site. Both of them address this problem.
One is for net neutrality:
https://petitions.whitehouse.gov/petition/maintain-true-net-neutrality-protect-freedom-information-united-states/9sxxdBgy
The other requests that the the broadband providers be classified as common carriers, which would require them to treat all traffic equally:
https://petitions.whitehouse.gov/petition/reclassify-internet-broadband-providers-common-carriers/4MrqLTlV
Signing the partition would show that people aren’t happy with the way the broadband providers are behaving.
I wouldn’t format either of the partitions at whitehouse.gov if I were you.
It sounds like you’re describing a kidnapping situation and/or a racket, and those are both clearly illegal. I like the word extort too, it works for this quite well.
Bob, Whatever happened to all this multicast streaming VoD providers were supposed to be using by now? Shouldn’t that cut bandwidth used by an order of magnitude?
[…] ← Level3 is without peer, now what to do? […]
Welcome back to AOL and CompuServe. That is exactly where today’s ISPs want to take us. They want us all in their walled gardens, where we can consume bits according to their business plans.
Nobody remembers history. The internet took off precisely because nobody owned it, where none of those proprietary walled gardens ever did or were ever going to. Today’s ISPs think they can have all of the money they’d like by skimming cash out of the old-style free-ranging internet. It won’t work, they’re killing the goose.
Common Carrier may be the only thing that could fix it.
That’s it, and what I’ve been saying all along, Bob. I already paid my Isp to transport those bits to me. It’s double-dipping if they also charge YouTube, NetFlix, etc.. for sending those bits I requested. I did not just purchase outgoing bandwidth from them, I also purchased inbound service, too.
If they refuse to upgrade and add more hardware, maybe these Isp and backbone providers should also stop signing up new customers, too? Otherwise the FTC or FCC should investigate them for deliberately over-selling of their pipes and misrepresenting the expected speed/bandwidth.
1. About 5 years ago, I sat in the Utopia offices in Salt Lake City whilst a bunch of experts I was lucky enough to share room with costed a full national redesign and rebuild of the backhaul, long haul, peering networks (call them what you choose) in both USA and UK. (I’m a Brit so having seen the US results, I asked that we replicate all the maths for UK too). I guess we should try to find the photos of the whiteboard. It was FAR cheaper than any of us expected and potentially hugely disruptive to established telcos.
2. Who needs an ISP any more? Get rid of all of them.
3. Let’s go back to Flat Rate Internet Access that we fought hard for in EU without any judgement from those soon to be non-existent ISPs (see 2) over what bits we are permitted. Cost of data transfer has always tended towards zero.
Certainly here in Australia, I think I can account for part of how ISP’s can continue to rack up huge profits, as it’s something that has had me intrigued for some time. Slightly off topic (not that far), but I’d really be interested in hearing if it’s the same in other parts of the world.
Typically we’re sold a monthly plan on a 12 or 24 month contract at $X/month that will include an allowance of Ygb for each month – and usually paid for in advance. Using more than the YGb/month will attract either a per Mb surcharge, or choking of the download speed. All of which is fine on the face of it – my problem (and anyone I discuss this with tells me I should probably have better things with which to concern myself) is that the disparity in the plans on offer means that typically customers will subscribe to a plan way in excess of their likely usage – meaning that in plan terms I will have unused capacity every month. Now while if I use more than the plan allocation, I will either be subject to an excess usage charge or have my service throttled, but there is no credit to me if I use significantly less than the subscribed plan amount.
The conspiracy theorist in me is convinced we’ve been suckered by the Telco’s/ISP’s into accepting a business model that has no parallel in any other type of business – certainly not for utilities like power and water which to my way of thinking are the closest analogies to the internet. You pay a charge every month (or quarter) for provision and maintenance of the infrastructure (pipes, cables, whatever), and a usage charge based on the volume of product you take from that infrastructure. The notion that all users will want to use their 100Gb allowance every month is probably as ludicrous as suggesting that I would use the same amount of water or electricity – month in, month out. So why can’t I buy data that way?
Am I missing something, or is it as I think – we’ve just been suckered by the rapacious Telco’s into a business model in which they will always win (and there it is right there – the answer is in the question!)
I decided to check my AT&T bill just now. I bundle Internet, T.V. and phone services. AT&T is charging me $24 a month for Internet. State, Local and Federal Government fees and taxes total $27. (That’s taxes and fees for all three services.)
I agree with Bob’s comments and think it’s a big money grab by the ISPs-in-question, but signs are they will be opposed by other industry heavyweights (Amazon, Microsoft, etc.). This will be an interesting influence-peddling case to watch.
I also want to mention that the worst ISPs might just be that on the basis of having a lot of customers which also implies a lot of complaints, i.e. the more customers, the greater the odds of getting a complaining one. I used to have Comcast cable internet and it was great, reliable, fast service, but EXPENSIVE at $67 per month. I switched to Frontier Communications fiber optic at $30 per month. It’d be great if fiber optic was like some public utility instead of private, but there it it.
It’s a good idea, but the FCC and the Justice Dept. are in on this, so there will be no action.
.
We will have to wait for Google to go over the heads of these assholes by putting up balloons or drones.
.
That might not be that far off.
[…] Level3 is without peer, now what to do?: I, CringelyVirtru Encrypts Your Email, Lets You Recall or See If It’s Forwarded: LifeHackerThis Was the Best Watched Solar Flare Ever: UniverseToday […]
I have a question about the Level 3 blog post. They show a lot of packet errors on a congested exchange point in Dallas — even during times when there is no congestion. What is the explanation for that? Are they arbitrarily dropping packets?
Give the responsibility for internet service to the U.S. Post Office. Seriously. They have a funding problem, because snail mail is being crowded out by electronic data delivery. Let them support the former with the latter. “Internet service is an extremely profitable business for the companies that provide it.”
.
Give the public at least a public option for internet access. Let the cable companies compete on content, not the data piping.
It *can’t* be symmetrical, by design.
The ISP’s AUP restrict users from providing services beyond their LAN. This means we can only be consumers of content and not providers.
Technical Restrictions
– use or run dedicated, stand-alone equipment or servers from the Premises that provide network
content or any other services to anyone outside of your Premises local area network (“Premises
LAN”), also commonly referred to as public services or servers. Examples of prohibited
equipment and servers include, but are not limited to, email, web hosting, file sharing, and proxy
services and servers;
https://www.comcast.com/Corporate/Customers/Policies/HighSpeedInternetAUP.html
I think Netflix should institute Operation Bilgewater, in which Netflix clients send them an equal number of bits (or saturate their upstream speed limit if that’s lower). Netflix can just pipe the data right to /dev/null.
We could also just make a screensaver that sends useless data upstream to Netflix, potentially turning the asymmetry on its head.
Coalition of 100 internet giants including Amazon, eBay, Facebook, Google, Microsoft, Netflix, Twitter, and Yahoo, took a stand against FCC on net neutrality:
“Tech Titans to FCC: Your Plan Sucks”:
https://www.theverge.com/2014/5/7/5692578/tech-coalition-challenges-fcc
Help me understand the economics that two people should pay the same for Internet service whether one consumes 1 GB per day or one consumes 100 GB per day?
If the low data user pays the same per day as the high data user is not the low data user subsidizing the high data user’s consumption? If you believe that is fair would you support flat rate pricing for packages so that whether a package weighs 1 pound or it weighs 100 pounds the postage is the same?
Keep in mind that your ISP doesn’t create any bits, just delivers them when needed. The actual cost depends on several factors: 1) fixed costs that continue even when you are not using the internet, 2) the cost of delivering bits at slow speed for those customers that can wait for them, 3) the extra cost of delivering bits at higher speed for certain services that contain so many bits the customer doesn’t want to wait all day, such as backing up or retrieving gigabytes to the cloud, 4) the cost of delivering streaming media, where the entire route from source to destination must be nearly instantaneous and late arrivals can’t be tolerated; this cost is even higher at “prime time”. All of those factors also depend upon where the customer is located since it’s cheaper to deliver to a large group than a small, geographically scattered, group. So technically our bills should look incomprehensibly complicated.
It is not too complicated for a mobile provider to charge for bits. Why is it OK for Verizon Mobile to charge for bits but unseemly for Verizon FIOS to do the same? Seriously curious to understand where the expectation of “unlimited bits” internet service came from.
The issue isn’t charging for bits but discriminating against certain bits depending on their source. This is what would happen if preferential treatment is given to bits whose originators (e.g. Netflix) pay an extra fee to a distant ISP (not their own ISP) for better, more reliable, and faster access to the customers of the distant ISP. Customers of the distant ISP would find their reception of all the bits they want would become less reliable unless the originator of those bits was paying extra to the receiving ISP for access to the receiving ISP’s customers. For example, your Netflix would work fine, but your Skype or free YouTube would suck. As a customer, you would have no control over the discrimination, no matter how much you were willing to pay. Your reception quality would depend on a bidding war among the big entertainment providers.
I don’t agree with your assertion that it’s OK that Verizon Wireless is charging for bits… End users are paying good money for those bits already…
Along these lines, can someone ask whether net neutrality ever existed at all? Akamai and F5 have been helping big corporations like Disney circumvent internet bottlenecks for over a decade now. Those who have had the money have managed to purchase faster delivery schemes for over a decade.
Could it be, then, that telecommunications companies are consolidating so that they can extort money not from the small guys, but from the big guys? Are Hulu, Netflix and others willingly submitting to the extortion because they see no other way out?
To be sure, the telecommunications industry is in desperate need of regulation because providing good service at a reasonable price for a reasonable profit is not good enough for them.
It would be interesting to see Level3 enter the consumer market to start raising the bar for the level of access consumers are long overdue.
If the ISPs are complaining that they are having to transport more data to their end users than they send back to the backbone provider, then the end users should just put in a small computer, e. g. Arduino, and have it simply transmit bits. Most people have wifi at home so it shouldn’t be a big deal to set up. There, now we’re even!
There is something profoundly unsatisfying and nauseating about stooping to the level of ISPs, but if you have a recipe, there is a good chance I’ll follow it.
[…] Cringely: […]
Simple really. Just send a notification to the end customer that the service is being blocked by their ISP. Jam their phones up. Peering ISP’s will quickly advertise that competing ISP’s are giving customers the “full internet”. If I cannot get to youtube, I’ll switch ISP’s that day.
Generally speaking, the phone company is much worse at YouTube than cable. The problem is there is no competition.
This is all about the payment model. There is no doubt that for example, Netflix is paying an ISP for network connectivity. I believe they are paying Level3 and I’m sure they are paying them quite handsomely. The issue is that a fair number of Netflix customers (who are also paying for Internet Connectivity) use Comcast. So Comcast gets nothing directly from Netflix to pay for the increase in infrastructure requirements to support the additional traffic that streaming video creates.
Keep in mind that Comcast is also a source of that same Streaming Video traffic so they are a bit hypocritical here complaining about video traffic when they are a source of said Video Traffic on other peoples networks because they make it possible for you to roam off Comcasts network and still stream their videos.
I agree with the author of the article.. Comcast is asking to be paid for something their customers are already paying out the wazoo for.
Truth is… Comcast made a HUGE mistake buying NBC the way they did… Big BIG BIGGG Debt and that is why they can’t afford to upgrade their network. And you know something… NBC is the only network that does not have a live streaming app for Android… so they can suck it as far as I’m concerned. I missed the season finale of Grimm last week because there was no app and I’m still pissed!
I think the big issue is that Level3 wandered into the CDN business making this whole discussion murkier than it should be.
Its no surprise that Akamai pays big money for ISP interconnects, but Level3 tried to undercut their pricing by using their previous peering agreements. Its actually incompetent management, expecting the traffic levels wouldn’t raise eyebrows.
> make their profit on the Internet because it costs so little to provide once the basic cable plant is built.
And this is the most important fact in the article. Few members of the public realize just how cheap it is to “move bits” around once the capitol investment of building the infrastructure is paid off. Actually moving the bits costs only the electricity consumed by the hardware, and is so small it effectively approaches free.
Re: “once the capitol investment of building the infrastructure is paid off”. But it’s never paid off. Upgrades are needed to handle TV.
I’m looking at my Backblaze status, and I’ve got over 100GB to get uploaded over Comcast’s crappy network. It will never happen. The fact that traffic is asymmetrical on Comcast’s network is entirely an artifact of a) the limitations of their coax cable technology, and b) the plans they sell their customers. I would love to be able to have an almost instance snapshot of my computers in the cloud, but that is not how the last-mile ISPs see themselves. They come from a world where almost all bits go one way.
Regarding this comment: “Short of implementing effective anti trust laws directed towards the ISP’s. The more viable alternative in my opinion would be a organized coalition of grass roots movements. Encouraging local governments to build their own back bones. Offering their citizens a competitive alternative to the major ISP’s doing business in their communities. The question is, can a grass roots movement grow to be strong enough to thwart the big money influences which created this mess in the first place? The good news it has been done elsewhere in this country.”
Folks…it *was* this country and its citizens that built the internet in the first place – NSFNet referenced in the article? That would be our tax dollars hard at work. So insult to injury…we built the service, they plugged into it, now they want us to pay *again* for something we built in the first place and are paying to use today.
True, we built the original, slow, internet, which was never designed to replace cable TV. So someone needs to pay for the infrastructure upgrades. The main questions are who and how. It could be all taxpayers, or all users of the Internet, or just those who wish to stream video, error free, at high speed, from the opposite side of the country, during prime time. It seems to me the issue is more complex than saying “we are already paying for it”.
In Israel there is an elegant solution to this problem. You generally buy “network infrastructure” (e.g. ADSL, cable, fiber) separate from “Internet provider”.
This way, the infrastructure provider has no saying on the routing of the packets going out of its network or coming back. The infrastructure sells bulk bandwidth, the Internet provider sells connectivity and if their peering sucks, people switch to a different provider.
The customer pays by the bandwidth (which is not guaranteed to be consistently high) – a customer with little needs can buy 500kbps while media addicts can go for the 100gbps.
That’s what we call “net neutrality”.
Reed Hastings: “[W]hen we ask [ISPs] if [Netflix] would qualify for no-fee interconnect if we changed our service to upload as much data as we download—thus filling their upstream networks and nearly doubling our total traffic—there is an uncomfortable silence.”
http://arstechnica.com/information-technology/2014/04/bittorrent-netflix-should-defeat-isps-by-switching-to-peer-to-peer/
[…] I, Cringely Level3 is without peer, now what to do? – I, Cringely The solution to this problem is simple: peering at the original NSFnet exchange points should be forever free and if one participant starts to consistently clip data and doesn’t do anything about it, they should be thrown out of the exchange point. […]
If ISPs are insanely profitable, please explain to me why I am struggling to keep the doors open and being overwhelmed by Netflix and Youtube traffic which I am not being properly compensated for delivering.
Brett.
Nice point.
Video content online is extremely hard to monetize. Essentially no one can pay anyone because no one wants to pay for video content being transferred online. But users love its on demand nature.
In what way are you not being compensated? Does your ISP provide free service to customers? I suspect you charge customers, so in your mind, what exactly are they paying for? And, symmetrically, why do you think that the vast money Netflix and Youtube have paid to *their* ISP is, I guess, too low?
Internet service must be content-agnostic.
[…] It’s so much more pleasant reading someone who knows what the fuck they are writing about reference ISPs, infrastructure, and bandwidth. […]
[…] soluzione ce la suggerisce il nostro lettore Andrea Paiola nel forum, che cita il blog I, Cringely. Secondo l’autore del post linkato dal Paiola, bisognerebbe tornare al concetto di peering […]
[…] Level3 is without peer, now what to do? […]
[…] I really like the simplicity of Robert X. Cringely’s solution: […]
Ariana Grande naked
[…]although web-sites we backlink to below are considerably not associated to ours, we really feel they’re essentially worth a go via, so possess a look[…]
Pinganillo para exámenes
[…]Sites of interest we have a link to[…]
motocourier
[…]Here is a great Blog You might Obtain Interesting that we Encourage You[…]
penurun berat badan tiens
[…]Here are a number of the web-sites we recommend for our visitors[…]
quality advertisement
[…]Here is a good Blog You may Obtain Fascinating that we Encourage You[…]
gümrük
[…]Every after in a even though we opt for blogs that we read. Listed below would be the most current sites that we opt for […]
emb gas
[…]that is the end of this post. Here youll come across some internet sites that we think youll enjoy, just click the links over[…]
overnight pet sitter naples fl
[…]the time to study or stop by the content material or internet sites we have linked to below the[…]
follow me
[…]very handful of websites that happen to become detailed below, from our point of view are undoubtedly very well really worth checking out[…]
cat sitter naples fl
[…]just beneath, are quite a few completely not connected sites to ours, however, they’re certainly really worth going over[…]
Green Smoke review
[…]here are some links to web-sites that we link to because we assume they may be really worth visiting[…]
reviews of Green Smoke
[…]The details talked about inside the post are a number of the top offered […]
Natural male enhancement pills
[…]we came across a cool website that you simply may enjoy. Take a appear when you want[…]
cheap jewellery wholesale
[…]Wonderful story, reckoned we could combine several unrelated data, nonetheless definitely worth taking a look, whoa did a single learn about Mid East has got a lot more problerms too […]
Digital marketing Indianapolis
[…]Here are a number of the internet sites we suggest for our visitors[…]
http://vxlo.com/artist-hit
[…]always a significant fan of linking to bloggers that I like but really don’t get a good deal of link really like from[…]
garage door repair
[…]usually posts some pretty fascinating stuff like this. If youre new to this site[…]
poncelet or poncelet brian
[…]Wonderful story, reckoned we could combine a few unrelated data, nonetheless really really worth taking a search, whoa did 1 master about Mid East has got a lot more problerms at the same time […]
DEA EPCS
[…]Wonderful story, reckoned we could combine several unrelated data, nonetheless genuinely really worth taking a search, whoa did 1 learn about Mid East has got additional problerms also […]
tempur-pedic mattress reviews
[…]we came across a cool site that you simply may love. Take a search if you want[…]