Sony’s huge PlayStation Network (PSN) has been down for a week now following the theft of ID and credit card data on some or all of the gaming and video entertainment network’s 77 million customer accounts. Readers have been asking for comment but I stay out of these things unless I have something new to contribute. That something finally comes a week into the crisis as gamers begin to wonder why the network is still not back in operation and speculate on what this all means to Sony? It’s a huge loss of face, if course, but beyond that the damage to Sony is minimal. And the upside for PSN members, including those involved in the many emerging class action lawsuits, is likely to be bupkes. Nothing.
Recent history suggests Sony’s likely gift to users as an apology for losing their personal data will be some period of free credit monitoring and a free month of PSN service. If that sounds generous you might be surprised to learn that the going price for wholesale monitoring from the big U. S. credit reporting firms is approximately five cents per account per month or $3.85 million if all 77 million PSN accounts have been compromised. The usual terms for a mea culpa of this sort are three months of monitoring for a total cost to Sony of around $10 million.
“It will cost them more to send the e-mails making the offer than it will to provide the service,” said a source of mine in the credit reporting industry.
If you are hoping for big bucks from a class action lawsuit, go back and read PSN’s Terms of Service you clicked on without reading when you first joined the network. As with nearly all such legal agreements, you signed away any significant right to compensation beyond the direct cost of the service for the time it is disrupted. Only the lawyers will make a dime from this.
That is not to say that Sony doesn’t takes the attack or subsequent outage lightly. No Japanese company would. But the fiercely proud corporation also hasn’t gone out of its way to apologize. No Japanese company would. A funny thing about Japanese business culture is the tendency to apologize profusely for absolutely anything that is beyond the control of the company or its executives. They’ll apologize for traffic, for bad weather, for someone else’s mistake, but if the company or its leaders have actually screwed-up they generally won’t say a thing, which is not at all good for Sony’s global image.
This outage comes in large part because Sony has been so aggressive against hackers, who finally decided to slap-down the electronics giant. This is not to argue that Sony shouldn’t defend itself, but it is to argue that Sony should have expected elevated attacks as a result of its actions. Maybe they did expect more trouble, but the fact that they were so easily compromised shows corporate hubris at a reckless level.
Now let’s consider for a moment why this outage is continuing a week after the break-in. Speaking with a few experts and reading the official Sony FAQ gives some insight into what may really be going on. Sony says it is investigating, but should an investigation really take this long? Can’t the server logs and other network data be locked-down in a few minutes and examined at leisure? Sure. So when Sony says it is investigating what they probably mean is they are trying to fix the problem, seal the breach, and make sure that particular gambit cannot be accomplished again. This takes time — hours of programmer time and dozens or even hundreds of hours of QA time to make sure the fix scales properly and will work under a full network load.
Sony doesn’t say this, of course, but that puts us back to the fierce pride part. While they can admit a break-in they find it very difficult to say they are putting locks on the doors that never had them.
But wait, there’s more! In the official Sony FAQ and also the Official PlayStation blog there is an amazing admission that the company really has no idea how many user accounts were compromised. They suggest that users “assume” their data has been stolen. Well, was the data stolen or not? That big unencrypted or shoddily encrypted file with the details of 77 million account holders either left the building or it didn’t, right?
Sony doesn’t seem to know.
This is from the official PlayStation blog:
Q: Was my personal data encrypted?
A: All of the data was protected, and access was restricted both physically and through the perimeter and security of the network. The entire credit card table was encrypted and we have no evidence that credit card data was taken. The personal data table, which is a separate data set, was not encrypted, but was, of course, behind a very sophisticated security system that was breached in a malicious attack.
Q: Was my credit card data taken?
A: While all credit card information stored in our systems is encrypted and there is no evidence at this time that credit card data was taken, we cannot rule out the possibility. If you have provided your credit card data through PlayStation Network or Qriocity, out of an abundance of caution we are advising you that your credit card number (excluding security code) and expiration date may have been obtained. Keep in mind, however that your credit card security code (sometimes called a CVC or CSC number) has not been obtained because we never requested it from anyone who has joined the PlayStation Network or Qriocity, and is therefore not stored anywhere in our system.
I love the part about it having been a malicious attack. Had the attack been less malicious, would less data have been lost? That is the sound of Sony whining.
When I discussed the attack with a friend of mine in the enterprise data security business he made an interesting speculation. “A really smart criminal would want to cover his tracks,” said my friend. “You can either grab the data and hope to slink away unnoticed or you can grab the data then destroy everything on your way out.”
In a worst case scenario, Sony doesn’t even know what vulnerability the crackers used to gain entry. Sony may be literally clueless.
The people behind this PSN hack didn’t want it to go unnoticed. They wanted Sony and Sony users to know they had been violated. And Sony’s apparent ignorance of just what was taken (possibly even how it was taken) plus the fact that the network is still down a week later strongly suggests the crackers may have thrown a few metaphorical hand grenades into the system on their way to Dennys for that celebratory Grand Slam breakfast.
There is another potential explanation to this story. What if Sony like many other large corporations replaced their experienced IT staff with an outsourced service of very inexperienced staff? I’ve seen this problem a number of times in the last year. Websites have been down for days with problems that should have been fixed in minutes or hours. When dealing with websites and security experience is important. If Sony outsourced PSN to someone without the right skills it could take weeks to fix it, if they even can fix it.
Sony said they turned to outside experts for help, so it looks like you are correct that this was beyond the level of their inside staff, but I think it is going too far based on the current data to claim that they were incapable of restoring the system at all. Only time and a lot more information will tell….
I wouldn’t read too much into Sony bringing in external experts to assist with the forensics. Since they conduct Credit Card commerce on this system, Sony would be required by their merchant agreements to engage outside expertise to perform a forensic investigation after an intrusion. Visa and Mastercard have certified security firms you are required to use in these circumstances. The card issuers don’t want to trust the investigation to the merchant since few merchants have the tools or expertise internally to perform a proper investigation and RCA.
I’m not saying that Sony did have the talent or controls to properly manage the security of PSN. Just that engagement of outside expertise is not evidence of incompetence. The fact that the intrusion occured at all, though, is obviously evidence that incompetence existed at some level.
I used to love Sony’s products, but the rootkit CD’s, infected USB sticks, OtherOS back peddling, data loss, etc have really tarnished their appeal. I find it hard to believe more people haven’t dropped them like a dirty shirt.
Maybe this is the final straw?
Have there been any claims of responsibility? I’m curious to know if it’s Anonymous.
Anonymous anonymously claimed they were NOT responsible AS A GROUP but allowed that individual members may or may not have acted against Sony.
I have a conspiracy theory. It was Anonymous but their “plan” backfired when Sony said personal data was stolen. Anonymous say it’s not their thing to steal data. Maybe that’s why Sony can’t find evidence to prove it, because data wasn’t actually stolen. Now, how can Anonymous admit to the attack without people thinking they stole the data.
my feeling is that the attack wasn’t just one hole in one machine or service, but more like a complete infiltration of their entire network.
they are probably looking at bare-disk re-installs on several thousand desktops and servers. perhaps even routers and firewalls.
they were so focused on copyright infringement and PS3 keys that they didn’t bother sniffing for rogue traffic on their lan. sony got completely pwned.
Fake? Maybe not. IRC chat log with the hackers: http://pastebin.com/m0ZxsjAb
The link below was posted at the Game Informer website. What does it mean?
http://lo-ping.org/2011/04/26/psn-hacker-chat-logs/
And just a few quibbles about the blog post:
“…and a free month of PSN service…”
Actually, the Playstation Network is already free. Not all accounts would have cc info if they just wanted access to the free network. In any event, a cc is not necessary and PSN cards are readily available at retailers such as Target and Best Buy.
Very disturbing indeed. So the credit card info was encrypted but I have to believe majority of folks use the same passwords for PSN as they would for other services, email, banking and so on. The unencrypted data is far more criminally valuable! You have to use strong passwords that are unique to each system and service. How is the difficult part. I use LastPass and the techniques found here -> http://hitechbrew.com/password-recreational-browsing/
guess what:
http://it.slashdot.org/story/11/05/05/1831217/LastPass-Password-Service-Hacked
I think this proves that “the cloud” is unworkable. A company even as big as Sony cannot think of every possible weakness in the network, so if you were to store data on the cloud, I would hope it is with the realization that it won’t be secure, it won’t always be accessible, in short, you would have been better off investing $10 in a USB key or SDHC memory chip.
“the cloud” has a lot of similarity to a VAX disk cluster set. it’s very easy to get working, immediate gratification, you look like a genius — with no effort but a command-line entry, you have grown your storage n times. hope you locked out users while you started the disk set.
data is, however, striped across who-knows-what in who-knows-how order. you may assume that any glitch anywhere is going to crash and burn the whole system. your assumption will most assuredly be right.
just Amazon CS2 customers about the past week. and anything in journal or in the process of saving or retrieval may be lost to even the best backup plan.
just about everybody has multiple attached data sets in critical infrastructure, hung on bags on bags on bags on the side of the machine with different colored tape, and it doesn’t take long for even the architect to lose track of the special magicks for each data set… security, tools, etc.
this week, you can ask Sony about how that works for ya.
moral seems to be, our toys and tools have become more complex than we can manage by parcel and silo. and there’s a maze of twisty passages, all alike, that nobody can figure out.
More likely: the corner suits decided to take bigger bonuses, and to finance same, “out sourced” to the lowest bidder, core functionality (what is more core than a company’s data????). Like the Bankster who crashed the economy, they’ll blame someone other than those who made the decisions.
I agree about the “cloud” – where exactly is your data & who can accesses it? If it’s everywhere there are too many holes that can’t be controlled or even monitored in a timely & secure manner.
Robert wrote “the fact that they were so easily compromised shows corporate hubris at a reckless level.”
Unless you – or someone you know – were personally involved in the attack and know a. how long it took and b. what methods were used then I fail to see how you can justify this assertion.
I can only imagine that Facebook must be scrambling to check their systems now. It’s only a matter of time until a data breach like this happens with them and then watch. THAT will be the real bloodbath.
It’s just a rumour, but several dozen Ars Technica readers are reporting fraudulent use of their credit cards which MAY be related to this incident
http://arstechnica.com/gaming/news/2011/04/ars-readers-report-credit-card-fraud-blame-sony.ars
Bob,
The fact that they turned to an outside company means nothing about their approach to outsourcing. Incident Response is a difficult field and requires specialists (say Mandiant for example) to come in and assist.
This is one of those “art” fields where book knowledge is one thing but experience and feeling work together with technical expertise to solve the breach.
And it looks suspicious (based on what I’m seeing through the security community) that those CCs were either not encrypted at rest or not encrypted when passed back to Sony. It also appears that Sony has a backup tool to forward all sorts of data back to the PSN even when users apply the standard wiping tools.
Not having a PS3, I can’t verify any of that, but it reflects badly on Sony.
“That is not to say that Sony doesn’t takes the attack or subsequent outage lightly.”
I think that should read
“That is not to say that Sony doesn’t takes the attack or subsequent outage *seriously*.”
Also “takes” should be “take”.
[…] Hard to trust the PlayStation Network when you have 77 million names, addresses, phone numbers, login credentials stolen. […]
An interesting read.
http://krebsonsecurity.com/2011/04/millions-of-passwords-credit-card-numbers-at-risk-in-breach-of-sony-playstation-network/
“… go back and read PSN’s Terms of Service you clicked on without reading when you first joined the network.”
This advice. like Sony’s advice in their email to PSN members “to login and change your password” is hard to follow… since the network is still down!
I think “clueless” is a spot-on description of Sony.
How long until every credit card number issued is in the hands of unscroupulous people? 2 years??
From what I’ve read in the chat log linked above, more like yesterday. Seriously. It looks like Sony used sub-par protection mechanisms (their claims notwhitstanding). The hackers got access to the table with user personal data (unencrypted) and the table with user CC data (encrypted, but it looks like the keys are readily available). This alone opens up the possibility of a huge wave of identity and CC theft.
But that’s not all. What Bob is speculating about is whether the hackers left the tables as they were. A hacker who was after the data itself would have. It would be in their best interest for the break-in to be discovered as late as possible (if at all), to maximize the time window needed to use the data they stole.
But it looks like getting the data was just a bonus. These hackers wanted the break-in to be felt. And if their primary motivation was to embarass Sony, I fear those tables are not the tables they once were. At the very least, they were completely wiped out. But if they wanted to get creative, they could shuffled the data rows or overwrite them with random information.
So the questions are, does Sony have backups, how old is the last one, are they readily available, do they have the skills and manpower to recover from said backups etc. I think I don’t have to remind anybody of the Microsoft/Danger/Sidekick fiasco and how fragile any “cloud” is without solid backup policies.
The frailty of the security defences is just half the story so far; stay tuned for the “our backups are unusable” story.
Bob, do you have any control over the advertising on this site? I’m seeing all adds for ‘usagc.org’ which appears (both on the face of it, and confirmed by web search) to be a scam. (Pay money to enter USA green card lottery, which you can actually do for free.)
Advertising? What advertising? Bob may not have any control, but you do.
Install AdBlock Plus or equivalent in your browser!
I’ll mention AdBlock Plus on odd days, and you do it on even days. Why is this such a mystery to people? It ain’t fresh out of the box.
maybe some people don’t like to steal.
Yea, as in not reading all the ads in the newspaper or not paying attention when ads run during your TV show. As an example of how silly this has become, I very good Windows 7 Forum (sevenforums.com) bans people for even discussing ad blocking (three strikes you’re out). So I’ll mention it here: “HOSTS file blocking”.
Maybe some don’t like being ripped off
Judging how Sony’s entire engineering of the PS3 was laughable in regards to security; it is highly likely that their internal networks were just as bad if not worse. The PS3 modders managed to completely tear down the PS3 and obtain every single encryption key, etc. The PS3 console is an open book.
Rumors are that there was a PSN Developer Only network, an exclusive location on PSN meant only for licensed developers and Sony workers. It was here where the breach happened. Security was rather lax and the hackers used their modded PS3’s to join that private PSN. Then they obtained passwords from the developers as apparently, nothing was encrypted. Then they logged into the internal systems as the Sony employees. Then only God, knows what they did and how long they were accessing the systems before someone finally noticed a problem. Apparently, they are running old exploitable RedHat servers, Apache servers, etc. that are all rather old versions with known exploits. ie… Sony has not been security patching their servers. The CC database may have been compromised, they just don’t know for certain. But if their encryption keys were just as easy to obtain as they were on the PS3, then it’s a safe bet to say at least some of their CC data was taken by the hackers.
Course it doesn’t help when you give an enormous gift to the hacker community like the ability to boot Linux on your console, then suddenly take it away!
Therefore leaving highly skilled hackers unable to run their own programs on the PlayStation. That ticks off these highly skilled people who reverse engineer for fun, they go and figure out how to hack the PS3 so they can run Linux again. Sony updates their firmware to stop them, they hack it again, and again, and again. Then Sony goes after the hackers using legal means. This just pisses the hackers off. If not the actual PS3 modding crew, the entire community of hackers. So some other individual(s) decides to take revenge or perhaps it’s someone hoping to deflect attention away from themselves.
Either way, it is not smart to take on the UberGeeks on the Wild Interwebs without first putting on asbestos underwear, a tinfoil hat and getting off the grid entirely! Just look at HBGary! You might as well stand up at DefCon, brag about your security while shouting insults at the crowd of hackers daring them to hack your servers! But just like HBGary, Sony’s security is very very lax with many loopholes and flaws. Either human error, laziness, incompetence, or being too cheap to spend on security measures are the cause. Add in a dose of arrogance and extreme pride and you have the makings of a spectacular fall from grace.
HBGary was social engineered and trust me just about any organization, even with a highly competent IT staff can get hacked that way. You get something going a little outside the norm for which there are no procedures and suddenly smart people start making mistakes and doing things they would never normally do.
Social engineering attacks are probably the hardest to prevent effectively and the least thought about in the industry today.
Not my area of expertise but I have read several articles asserting that user passwords should not be encrypted and stored. Passwords should be hashed and only the hashes stored. That approach might have saved Sony some trouble.
Your point about the length of the duration spent fixing the issue makes me wonder if it even “can” be fixed (short of an updated for all connected games).
What if the design of the network is inherently flawed and fixing it would break compatibility with shipped software?
My guess would be that they either; don’t know how they system was initially compromised or don’t have a *good* fix. One of the problems was the zero cost transactions the develop consoles could do. They may have a problem with end point identification which they can’t come up with a way to fix that is also completely opaque to all that software out there floating around on read-only blue-ray disk.
Sigh.
“A funny thing about Japanese business culture is the tendency to apologize profusely for absolutely anything that is beyond the control of the company or its executives. They’ll apologize for traffic, for bad weather, for someone else’s mistake, but if the company or its leaders have actually screwed-up they generally won’t say a thing, which is not at all good for Sony’s global image.”
Citation needed. Maybe it appears worse for you because you live in the Deep South? Here on the west coast I’m constantly surprised at how *similar* the responses of Japanese companies are to what I’d see from a company here.
Especially once you get past the language. The vast majority of American responses I’ve seen to press releases from Japanese companies have been from people reading only paraphrases of overly-literal translations. I’m getting tired of reading American commentators talk about the precise meaning of Japanese business apologies when they wouldn’t know one if it walked up and punched them in the face.
One fact has become painfully clear to Sony: you don’t say you know what happened unless you REALLY know what happened. I understand this is hard for American media to figure out, given that it’s built almost entirely out of 2-second sound bites, photos of celebrities, and suggestive innuendo (IS SONY PURE EVIL? ANSWER AT 11), but the rest of the world doesn’t seem to have such a problem with people saying they don’t know something they don’t know.
[…] 2011 by SulkaThese are things I’ve found interesting between April 25th through April 29th:Cringely on Sony’s Response to the PSN Fiasco –Real time vs. slow time – and a defense of sane work hours – (37signals) – […]
if this was an attack from the outside. why are they moving the hardware?
http://arstechnica.com/gaming/news/2011/04/sony-kept-some-psn-data-encrypted-and-is-physically-moving-hardware.ars
Yeah, your description of “Japanese business culture” is highly stereotyped, vaguely racist, and factually incorrect. I see Japanese CEOs apologize (and even resign as an apology) quite often, when the problems are their fault. And yes, I’m fluent in Japanese (and even work occasionally as a translator) so I know what they are really saying.
Perhaps Bob’s point is “resign as an apology” is more Japanese than American. In America resigning without compensation only happens in cases of deliberate criminal acts…mistakes are part of the risk of doing business.
Is it time for Sir Howard Stringer to commit seppuku?
[…] At I Cringely – Cringely lays out why the security breach at Sony may be worse than we thought […]
[…] Cringely Comments on Sony […]
Getting a tad racist here aren’t we? Sony did apologize, contrary to the “Japanese” way, that this article asserts. They had top executives, even bow 45 degrees, in front of the entire press corps. Certainly more than a written press release.
I mean to blame one culture or another over hacking is quite simply logically flawed. It isn’t even two years since the Heartland and Hannaford hack that compromised 130 million actual credit and debit cards. TJ Maxx had 40 million cards stolen, in comparison, 10 million credit cards that were encrypted by Sony certainly doesn’t justify racial sterotyping.
Not to say that you are a genius (we knew that already) but Sony is, indeed, offering identity theft monitoring.
http://games.slashdot.org/story/11/05/06/033251/Sony-To-Offer-Free-Identity-Theft-Monitoring
Thanks Mr. Cringely for your take on this fiasco, I hoped you would have an article on it, but I missed it till now, darn it! Although more information has become available, I think you have the right idea. Supposedly Sony had outdated versions of Apache running and shoddy or next to no firewall if this is to be believed. I guess I wouldn’t be shocked that a huge company supporting 77 million users on their network would not have proper security, oh ok I would be floored!.
(https://www.joystiq.com/2011/05/05/psn-servers-were-unpatched-and-had-no-firewall-installed-secu/)
I don’t know if they were outsourcing, as some have mentioned in the comments. Their datacenter was in San Diego, though.
My business card have been stolen ! Thx Sony !!!
shell bracelet
Developed in collaboration between legendary music producer and artist Dr. Dre, engineers from Monster Cable and renowned industrial designer Robert Brunner. The Beats by Dr. Dre Studio headphones allow you to experience music the way the artist wants you to. These high definition headphones are precision-engineered to reveal the full sound of today’s digital music including the most sonically demanding rock.
When it comes to information there is no such things as conspicuous consumption and none of us are ever information-rich enough.
Qr Code…
[…]I, Cringely » Blog Archive » Sony may be clueless in PSN hack – Cringely on technology[…]…
What a huge failure, I wonder how much Sony lost in this fiasco.
youtube porno francais…
[…]I, Cringely » Blog Archive » Sony may be clueless in PSN hack – Cringely on technology[…]…
[…] Hard to trust the PlayStation Network when you have 77 million names, addresses, phone numbers, login credentials stolen. […]
rumblefighterhackshackslataleogplanethackshackshackshacksprivateservermaplestorydragondragonnesthacksbossmode…
[…]I, Cringely » Blog Archive » Sony may be clueless in PSN hack – Cringely on technology[…]…