This is my promised column on data security and the Internet of Things (IoT). The recent Dyn DDoS attack showed the IoT is going to be a huge problem as networked devices like webcams are turned into zombie hoards. Fortunately I think I may have a solution to the problem. Really.
I’m an idiot today, but back in the early 1990’s I ran a startup that built one of the Internet’s earliest Content Distribution Networks (CDN), only we didn’t call it that because the term had not yet been invented. Unlike the CDNs of today, ours wasn’t about video, it was about the daily electronic delivery of PDF editions of newspapers and magazines. Canon told us that if the New York Times, say, would make a PDF version of its daily paper, the Japanese company would give an ink jet printer to every electronic subscriber, making their money solely on replacement ink cartridge sales. Communication would be between the CDN and printer with no PC involved. It was effectively an Internet of Things, circa 1994. Obviously, we failed, but learned a lot along the way.
Our network was called Pronto and had a few features so far ahead of their time that they still aren’t generally available in other products. It was my idea and general architecture but the actual network was designed by Paul Tyma before he built his own products that include Dash-O and Dotfuscator at PreEmptive Systems, a total redesign of Gmail for Google, Home-Account (my mortgage startup from 2008 — great timing with that one), Mailinator, Refresh, and most recently whatever they are doing at Lendingtree, where Paul is now the CTO.
Pronto was designed as a global delivery system using massive numbers of simple networked devices that worked together to deliver the newspaper no matter what. To Pronto nuclear war would have been a minor inconvenience.
Does this sound to you like a bot-net? That’s exactly what it was only we never thought of Pronto being used for evil. In fact I can argue pretty strongly that we designed evil out of it completely.
What made Pronto unique for the time was that it made only the most primitive use of Internet infrastructure, replacing services like DNS, for example, with something similar but different and unique to Pronto The system had publishers and clients but everything in-between was a peer-to-peer network where identities didn’t matter so much as proximities and loads determined by ping times using a bastardized multicast protocol. We didn’t even use IP addresses in the sense that Vint Cerf would have recognized them. If we could use today’s vocabulary to describe what we built 22 years ago, it was a Software Defined Network (SDN).
Part of the technical inspiration for Pronto came from a conversation I had with Roger Boisvert, a Canadian who was also a pioneering ISP in Japan. Roger ran Global OnLine (GOL), an early broadband ISP in Tokyo that specialized in English Language support. GOL supplied our test network in Japan.
NTT, Japan’s largest telco and ISP, was an investor in Pronto, but they couldn’t provide infrastructure for less than $75,000 per month, they said, because of regulations. Roger hosted us for free.
One day over lunch Roger explained his novel method of containing half a dozen Yokohama customers who used vastly (often 100X) more data than the average GOL member: he put each in his own bandwidth-constrained Virtual Private Network but never told them. Each of these customers got all the data they were paying for and no more. I thought it was brilliant so I simply applied the principle globally to Pronto! which became a VPN the size of the Earth.
We could do the same thing today with the Internet of Things. There is no reason at all why the IoT has to share address space with IPV4 or V6. The point is networking these things together, not networking them to mess with CNN or Facebook. As an SDN the IoT could use a radically different addressing scheme along with packets unrecognizable to most NICs and all built to overlay the regular Internet using the same fiber and routers. The bits would still flow through the same network, but it wouldn’t be evident what they were for until they reached their final destination, which would be a relatively rare gateway between the two networks.
Key here is the idea that the IoT has to be dirt cheap so that means microcoding the network in a way that’s super-cheap to build in volume but inevitably super-expensive to change (or co-opt). I’m not saying it would be impossible to turn a Pronto-ized IoT into a malicious bot-net, but I am saying it would be a lot easier to find your bots somewhere else.
Normally we’d say “the bad news in this is that everything currently in use for IoT has to be thrown away” but given the dire possibilities for data security of current IoT hardware the scenario shifts into an OPPORTUNITY to replace all six billion IoT nodes operating today, because doing so saves the future. And what hardware industry doesn’t want an opportunity to be paid again to replace 100 percent of its already large installed base?
Thank you, sir. May I build you another?
Not great timing -I’m afraid- with this one either – the IoT bubble is already pretty defunct. Next ones will most likely be the social media and Big Data, now that the elections have proven that they are both useless and a waste of return on investment (but I guess that you already are accounting for that in your predictions for 2017 – which hopefully you will publish soon!). Also, hope that you get well soon from the cataracts problem!
Also (hope that I am not asking the obvious here), but the whole point of the IoT, at least for home users (industrial users have totally different needs and much better security… I hope!), is that you let the users interact with the IoT devices from their smartphones and tablets (say connect your phone to the IoT lightbulbs), which means that they would need to share address space with IPV4 or V6 at some point, right?
“industrial users have totally different needs and much better security… I hope!”
I assure you they don’t.
I don’t usually post here, but I started reading the comments and this comment section can be a big hard to follow sometimes.
I don’t see IOT as being anywhere near dead. I work with quite a few customers who are coming up with all kinds of zany ideas for “devices” or “things.” To me, it resembles a lot like the early days of the Internet where people were creating solutions where no problems existed. Of course, those were crazy times and investors would pour millions into anything that had “.com” in their name.
I actually prefer the name “IoE” (Internet of Everything) because a lot of these “things” don’t necessarily need to be devices at all. Yes, I’m thinking “containers” in a very broad sense. A typical dual-socket server today could run 1000’s if not 10’s of 000’s of “containers” that at some point, will be monetized.
Just a primitive example: I’m not Youtube, but I want to have something that kind of resembles YouTube. I’d just need to design everything around using some CompanyX’s container — and it’ll (in theory) work better because CompanyX has already taken care of security, optimization, scale, etc.
I’m sure there will be some new buzzword that will eventually replace IoT, just like old timers like me laugh when customers move entire VM’s to AWS. 20 years ago, we called it “co-hosting.” 🙂
Currently most such systems have the IOT device connect to a central control system and the phone app also connects to and sends commands to that central system. All you’d need to do is put the IOT side of the network, between the light bulbs and such and the control system, on the secure network.
Don’t be so sure about Big Data failing in the election. Maybe the Clinton campaign was just not as competent at it, while Trump’s campaign was secretly good.
Also, Clinton never spent much in the closes states, pursuing a blowout win in Arizona and Ohio and Georgia. Spent more money pursuing the single electoral vote in Omaha than Michigan and Wisconsin COMBINED.
On top of that, we see Trump bought more ads in those states by a large margin, despite Hillary raising 5 times more money. Perhaps it wasn’t the failure of BIG DATA, but BIG CONSULTANTS, who pocketed a large share of the billion+ that was raised.
internal clicqueishness and sense of superiority, in our experience with the Clinton folks in our state, had a lot more to do with Clinton’s loss than the amount of Big Data they had. BD is only as good as how you use it. there were no signs, no flyers, no door hangers, no coordinated campaigning with the state and district candidates. Tip O’Neill said, “All politics is local.” all we got out of the Clinton campaign were two hoity-toity closed fundraisers for the near-millionnaire set. at least Trump bothered to set his plane down at the #2 terminal and let thousands of folks clog the main roads and saunter in to cheer.
where Obama’s Big Data went was in organizing down to the neighborhood, and getting feet on the ground to knock doors. Clinton just used it to fundraise.
moral… pretty things aren’t really admired unless you share them with others. drool over it all you want with the closet door closed, it’s a hobby. let everybody in on it, it can even be a business.
Are you saying you expect the number of IOT devices (webcams, home automation devices, drones, vehicle telemetry systems, etc) to fall? That seems unlikely.
In general, our plan today is to defend against the attack yesterday – while your idea sounds good, I suspect that one bad apple in the infrastructure would easily spread across the rest of the network. It’s hard to defend against that when your network is built by multiple vendors for the lowest cost possible.
In any case, the recent IoT DDOS only worked because we tend to reply on a single point of failure, the DNS addressing system – if DNS were distributed wider then the IoT attack would have been relatively ineffective.
Sounds like “security” by obscurity. Someone will find a way to poison this too. And if there’s no way in, how would it be useful?
Robert is really only addressing the DDOS problem we’re facing now. The security of the actual IoT isn’t directly addressed, and controlling access to / use of the IoT will become a huge economic issue as the IoT starts to do important things. But even though DDOS as we know it would be addressed, loading attacks on critical gateways would still be possible, changing the nature of DDOS attacks.
The real solution is accountability: where does this packet come from? That will be the real challenge/opportunity.
Based on your previous (and present) record of over committing and then never delivering, this project will never happen.
It’s nice to see our bitter Mr. M. poke his head up out of the emotional toilet he inhabits – and fart in our general direction again.
I’ve personally known a few guys like Mr. M. … Sad lonely sods, the world is simply not worthy of them. Everyone else is always the bad guy, and it’s their job to prove it. *They* never did anything wrong, or made a mistake – and if they did, it was certainly someone else’s fault !
Thanks for sharing, Mr. M. In return, I’d like to share something with you : A nice short story called “The Canterville Ghost. https://www.eastoftheweb.com/short-stories/UBooks/CanGho.shtml . Enjoy !!
Maybe you should pop your head up and investigate the situation before you respond. If you would actually look at the postings on the kickstarter page, you would see that the Mineserver project was advertised as a 21-day project that only needs funds for a custom cases before it could be shipped “since all of the engineering was already done.” There were also promises they the project would never let the backers down. Since then, the project is now over a year late and there have been numerous promises of “we realize that we haven’t been communicating but that will never happen again.” The backers only seem to get an update when we raise enough stink on this blog to get his attention. Many of the backers on kickstarter project are long time backers of numerous projects who realize that problems happen but none of us have ever been a backer of a project with so many lies from the project and so little feedback. It has been over a year since the project has responded to any technical questions about the project. Of course, there was a posting in this blog a while ago about “those folks on the Internet can’t be satisfied.” For being run by a professional communicator, the communication on the kickstarter proect has been pitiful.
Most computer computer projects are total failures. Numerous studies on that issue and anyone honest in the business knows this all too well ( even if not publicly stated ). I myself, am always somewhat surprised when something involving a computer actually works perfectly as intended. So it is sort of boring complaining about it and certainly detached from reality to believe this is unusual. Sadly, we seem to be living in times of increasing detachment.
Re: “I myself, am always somewhat surprised when something involving a computer actually works perfectly as intended.” No one would disagree with that due to the word “perfectly” but it applies to life in general and the universe as a whole, not just computers. If the kick-starter is a failure or going to be delayed indefinitely, that fact should be communicated to the backers.
I am out of my depth here, but I wonder if IoT devices built on Harvard architecture machines would be immune to takeover. If all data from a network were constrained to data memory, and all programming were contained in ROM based program memory, it shouldn’t be possible to alter the program. Data could not become machine instructions. Would that help?
I thought the same thing. I believe a harvard architecture would make it impossible for problems like buffer overruns from allowing the injection of new code. Many microcontrollers are designed with that architecture so separating data and code wouldn’t require any special effort.
What happens when cheap-webcam-company goes out of business and takes their VPN with them? I’m left with a doorstop, or a house full of doorstops.
I think you have a good start towards a solution. From an Internet users point of view we’ve forgotten about the concept of “local” area networks and “wide” area networks. We used to use different network protocols. Some could be routed some could not. With the Internet and TCP/IP everything we do is via a wide area network. There are IoT devices I could put in my home that should NEVER have access to the public Internet. Having an approach that restricts the access of the next generation of Internet devices is not only a good idea, I think it could become a necessity.
This sounds like the explanation for the Minesever delay. Was the Mineserver Kickstarter campaign really used to fund this endeavor?
Wait… we have to scrap all our cars and all our IoTs? Any time the solution is “start over” it’s a solution that lacks creativity or any appreciation for the real world.. look how often Hollywood reboots something because they don’t understand how to work with what they’ve already been given.
We can do better.
IoT insecurity is an economic, not a technical problem. As Bruce Schneier puts it, neither the seller nor the buyer care about the collateral damage, which is why he advocates for government intervention to curb the externalities caused by market failure, e.g. liability laws. What you describe would require router manufacturers to add the VPN functionality, IoT device makers to have the device negotiate its jail with the router (presumably using something like UPnP or 802.1X, which themselves are a major source of security problems). Given neither the router manufacturer nor the IoT maker has any incentive, and the customer who buys both doesn’t care, there isn’t much likelihood of it happening without a government mandate, and that is also unlikely to be forthcoming:
https://www.theregister.co.uk/2016/11/16/experts_to_congress_you_must_act_on_iot_security_congress_encourage_industry_to_develop_best_practices_you_say/
Routers are unaffected and need not be changed for a Pronto-like overlay network to exist. Bits are bits.
? !? Hubs could be bitwise, tho’ more likely framewise. Router hardware/firmware runs a full Internet Protocol stack, implementing Layers 1 through 4 (they are responsible for TCP handshaking). “Pronto” may be compatible at Layers 1 and 2, but not at Layer 3 or above, unless I have completely misinterpreted your description of what kind of thing it is.
Bits might just be bits but protocols are a totally different layer in the OSI model. For instance, if a router is set up for TCP/IP and you give it IPX/SPX traffic then nothing is going to get through. Sounds like your “magic overlay” is sort of TCP/IP encapsulation. Hope you aren’t counting on jumbo frames.
Actually, routers, that’s a good point.
Sounds like what you’re describing is a device and gateway system, a standard IoT configuration. And the problem is that at some point, something connects to the Internet. So rather than infect your light switch, they just infect your gateway/VPN device. Big difference.
Routers have the same issue. For such cheap and simple-looking devices, they are full of services and complexity. And third-party routers hardly ever get updated. At this point, I think most people’s home security is better served by just using the modem/router supplied by their ISP. The ISP will update the router’s operating system transparently, or be responsible themselves for the consequences.
I do not see an equivalent authority for IoT. I imagine if somebody really wanted to be serious about it, they would have to invest an incredible amount of money to build an ecosystem by brute force.
My point was that Bob started off describing Pronto as something that could not be an “overlay network” over an IP stack, but instead something more like the proprietary RF mesh networks that are used to read/control “smart” electric meters, especially in terms of adaptive routing. “…We didn’t even use IP addresses in the sense that Vint Cerf would have recognized them….” In order to be transparent to an IP router, Pronto would have to use IP addresses in the sense in which they are standardized.
Re: “The point is networking these things together, not networking them to mess with CNN or Facebook. As an SDN the IoT could use a radically different addressing scheme along with packets unrecognizable to most NICs and all built to overlay the regular Internet using the same fiber and routers.” I don’t understand. It seems to me the existing routers and infrastructure would just drop the bits that have no recognized protocol or packet structure associated with them. The key may be your reference to SDN, but that Wiki failed to explain it to me.
I noticed Bob also said “Pronto…became a VPN the size of the Earth”. So I guess he means the data stream going through the routers wouldn’t all be gibberish, but just the VPN-like data within the IP packets needed to route it through the Internet. In that case it would have been less confusing to dummy me with no mention of SDN or Pronto. Phones and computers would also join the VPN to interface with people, so just those devices would remain vulnerable, which they are anyway. The only new problem may be one IOT-VPN infiltrating another IOT-VPN, possibly through the manufacturer’s website. How secure are VPNs anyway?
I’m reminded of the axiom “Those who don’t learn from history are condemned to repeat it.” Most IoT devices are equivalent to a Windows 3.1/NT PC circa 1996: an architecture designed for a closed, isolated existence suddenly given a connection to the entire world, so poor access control, open defaults, difficult to update, purchased and operated by ordinary consumers who are unprepared to deal with the consequences (nor should they need to be). It’s like we’ve invented a new class of automobile with with no seat belts, crumple zones, airbags, or antilock brakes.
How about putting IoT devices behind a reverse firewall? Only allow the IoT device to talk to its manufacturer (or whatever small set of URLs it needs to talk to) and block everything else. Even if the device is compromised–and security will never be perfect, so it will happen–the device wouldn’t be allowed to run amok around the internet. This can be built into home routers–or the ISP’s edge device. We would need a good way to distribute profiles for the various devices that are out there that define who they should be talking to. Manufactures could do that themselves, which would be a good thing since it would allow them to CYA in a sense.
As Fazal Majid says above, the problem is not to invent some clever solution to IoT security, the problem is to prevent people from deploying stupid solutions and insecure devices. The devices used against Dyn had the worst imaginable security short of none at all: the oldest protocol on the internet, Telnet of all things, with hardcoded passwords, and no way to update the device when the inevitable bugs are discovered.
My own proposal is to mandate forced obsolescence: all devices must have a registered universal ID, current time and an expiration date no more than 1 year from build date encoded in the device’s TCP and UDP header options. Universal device IDs would be administered by a decentralized organization like the one that manages unique ethernet IDs, and built into the system ROM by ROM chip manufacturers. Any session from a device beyond its expiration date or with an invalid ID is eligible to have its packets dropped by any router along any communications path.
This has the economic incentive of providing a continuing revenue stream for vendors as customers replace outdated devices. It won’t solve the problem totally, but should at least avoid a cyber-Kessler syndrome where the internet collapses as the legacy of forgotten, untraceable, corrupted devices grows to overwhelm devices that are trying to do useful work.
Mandating people replace their gear – particularly if that gear is integrated into everything they own is going to be a very hard sell. A better solution would be to build better IoT devices that allow the firmware to be updated periodically to address the problems mentioned.
Yes, and now bad would that be for degrading the earth’s environment. Are you kidding me?
Good to see that Roger Boisvert is remembered, a great guy and, as you note, one of the pioneers.
As you probably know, after dinner at a restaurant in Los Angeles he and a colleague got lost and stopped the car to ask directions. A guy leant in the window and shot Roger in the head.
Utterly senseless and tragic
The value of IoTs isn’t so much their functionality in and of itself. The value is in providing that information or function to devices that are on the network that people use – such as your cell phone (need to open your garage or front doors, turn on the lights?). That is where the weakness is. If any of the nesting dolls that make up the chain of functionality in a computer – be it microcode, firmware, operating systems, or applications are compromised – then *valid* commands could be sent to IoTs – commands that are designed to take advantage of weaknesses in the IoT’s software stack – or just cause havoc with the IoTs themselves (e.g. turn off the brakes on 1 million cars). Network security is an end to end proposition. Unfortunately everyone has been focussing so much on 20% border security problem at the expense of the 80% zero day bugs problem. Business has it backwards – but only because they are thinking short term. Fixing that problem will be costly, but given how the stakes have been rising – I expect the crossover point between ‘too costly’ and ‘saves money’ to be approaching faster than we want to imagine.
There are still packets and IP v4 or v6 addresses whether it’s IP or UDP. The real problem lies in the “open” nature of the Internet where everyone keeps well known ports open at the common port numbers. And the fact that we all use the same software to listen and accept and all the weaknesses of that software is well known and exploited. How about a single “well known” identity port that you connect to and present your “identity” (your certificate, validated through a Certificate Authority), that “by law” must identify a “real person or organization”. If you don’t have that we not only “not let you in”, we block future requests both for your IP addresses at our router, we also block you up line at our gateway router. I realize that “by law” would mean international law, but I think it’s simpler than that. If CA’a require a deposit equal to the connections a user can access at a time, then online spamming is done. Legitimate business can pay the deposit (say $1000, $10,000, $100,000) and have access and forfeit it if their “identity” is used in a DDNS attack. It’s clear to me that Bob C. is a reporter and not a programmer with this and other articles. At it’s core, there are packets and some type of address that routes information between computers. Without a proxy up front to verify identity, DDoS attacks will continue to happen, and the best we can hope for is some smart algorithms to find and stop the “bad actors”. The network administrators can only guess and a huge DDoS attack by by all your IoT objects in your network could render you as a “bad actor”, unable to access the Internet without your knowledge or approval, because you let your refrigerator and light bulbs have access. The “old Wild Wild West” Internet can still continue to operate, just legitimate businesses and users will “identify themselves” before doing anything meaningful, like online shopping.
My interpretation of Bob’s idea is that he’s not trying to solve the DDOS attack problem in general, just the direct contribution from IOT devices. His devices only communicate over their own VPN only with a central hub, which in tern communicates over the internet to reach people. The hubs can still be turned into zombies, like PCs can today, but the IOT devices themselves don’t know about Internet protocols. If one hub controlled 10 devices, we could have 1 DDOS zombie instead of 10.