My e-mail inbox this morning contains 118,306 messages totaling about seven gigabytes. I really should so something about that but who has the time? So I keep a lot of crap around longer than I should. I have, for example, every message I have sent or received since 1992 when I registered cringely.com. Those obviously occupy a lot more than seven gigabytes, though interestingly enough the total is less than 20 gigs. My storage strategy has been a mixed bag of disks and cloud services and probably stuff I’ve forgotten along the way. So I’ve decided to clean it up by standardizing on Microsoft’s OneDrive (formerly SkyDrive) cloud storage service, just relaunched with its new name. I need about 30 gigs of storage right now but I don’t want to pay for anything.
No problemo.
Microsoft gives away free OneDrive accounts with seven gigs of storage. If you save your mobile pictures to OneDrive they up that to 10 gigs. If you get a friend to sign up they’ll give you another half a gig of free storage for each friend up to a maximum of five gigs.
So I signed-up for the free 10-gig photo plan, created two more e-mail addresses on my mail server then recommended those accounts sign up to OneDrive, too, which they obediently did. So in a few minutes I had turned my OneDrive into a virtual ThreeDrive and exceeded my 30 gigabyte storage goal, making it to a total of 31.
This all requires some management on my part, but it’s just writing little scripts to store, find, or restore files or the contents of files. Everything is encrypted of course, so there’s nothing for Microsoft or the NSA to read without a lot of effort.
How can Microsoft afford to do this? Well for one thing I doubt that one user in 100 would bother to replicate my effort. For another, Microsoft is probably grateful for the hack because I wrote about it and helped promote their name change. It’s a win-kinda-win scenario.
Now what to do about all my video files? I have those on more than 30 external disks, some of the old ones only 1-2 gigabytes in size while others are up to two terabytes. So far I’m chugging through the disks one at a time on an old Pentium 4 box running Spinrite 6 from Gibson Research — still the best file recovery program you can buy. Like all the software I use I paid for my Spinrite copy, which causes Steve Gibson to comment every time we communicate. According to Steve I may be the only blogger who pays. Silly me.
Having a dedicated Spinrite machine makes a lot of sense because restoring a really corrupted disk that’s been sitting since 2005 or so can take up to a week of continuous churning. I’ll eventually move it all to some kind of Network Attached Storage, this time a multi-disk array with clever self-maintenance routines copied across to a similar box with my mother-in-law back in Charleston.
I wonder if the NSA knows about her?
One more thing: if this column seems to contain a lot of fluff that’s because I’m madly trying to finish my damned IBM eBook, which turned out to be not at all the lark I expected. I’d forgotten how hard books are to do. But this one is nearing its end and includes quite a bit of shocking new material never published before. I like to think it will have been worth the wait, but you’ll have to decide that.
Hey Bob, I went through some consolidation earlier this year and had to learn a lot about low-end NAS. In my case, I have lots of data that has to be kept on an encrypted file system, and that affects the choices. Briefly: Seagate BlackArmor was a steaming pile of crap that I wound up returning, Synology DS413 I’ve been very happy with, QNAP had a model very comparable to Synology and my choice was made based on some very very minor points.
What encryption software do you use to protect you in the cloud?
I don’t think I’ll answer that question directly. The issue with cloud storage is that it generally uses encryption over the wire (https) but not in the data center. That’s how the NSA was hacking Google and Yahoo, remember. Each has moved the SSL decryption stage deeper into the data center but I still don’t think they encrypt at the disk (correct me if I am wrong, Googlers and Yahoos). What I’ve done is simply encrypt the file BEFORE I send it over the SSL link so what’s decrypted on the other end is still an encrypted file. There is plenty of software that can perform this function, I just don’t want to blurt out a particular product name, thank you, just in case GCHQ is listening.
Encryption at rest is getting quite common, but all it does is protect you from dumpster divers.
For a file system to be encrypted, there must be a real time way to decrypt it — also known as an “attack vector.”
You’re point is…?
It’d be more like the marketers in Redmond to name it “Microsoft OneDrive for the NSA Cloud Capture System”.
Yo Bob…what about your Mom-in-laws PogoPlug? Too expensive for you?
I want to replace the PogoPlug under Mimi’s guest bed with a RAID device of some sort that makes it look like a contiguous space and offers some error correction and fault tolerance. The PogoPlug is doing fine and I could just put bigger drives on it but I want something fancier.
You could put Arch Linux on the PogoPlug and do as you will. It’s not Ubuntu easy, but it’s not terrible.
Another option is Amazon S3 and linux s3cmd tool. I use this for all my photo media right now. Like you I have ~30 gigs, and costs equal to a cup of coffee each month.
If you then set up the bucket to archive to Glacier, you would be spending around $0.30 per month.
(Of course, you might already be doing that and you just buy cheap coffee.)
Tape (which is what Glacier is probably using) is an extraordinarily cheap medium for long-term storage; it’s just the cost of the drives which make it expensive. I don’t know why we haven’t seen other companies offer this kind of service to amortise the cost of the drive over lots of customers.
Sounds like what you need is a ZFS NAS server. Have someone build you one. They are frik’n awesome. Google: FreeNAS Project, for more details.
FreeNAS is the code running on my cheap Buffalo NAS box. You are right, it is quite good.
Backup has always worried me. I’m pretty sure my wife would kill me if i lost our photo collection.
For the amount of data i have there are some pretty reasonable solutions online and with dropbox doing autosync from Iphoto it may well will when i finally pull the trigger. The problem is the inital data upload. I have pretty unreliable internet and i wouldn;t even like to guess at how long its going to take to upload 200Gb of photos.
Now to my real point. I know why MS are changing the name this time (legal issues with SKY TV in England) but surely all these name changes are killing any chance they have of building an online brand.
One drive was previously Sky drive, before that Live drive and i’m sure there was another one.
Hotmail changed to Outlook although for a time i’m sure they were pushing live mail.
Bing, MSN and several other online entities all dilute the branding
I really though the xbox would be the xbox8 to match windows and phone branding but that went out he window and One is the new marketing point.
Xbox Live and Xbox video and one video and xbox music. Its just all really confusing.
Look at their competitors
Google, Google mail, google maps, google calenders google plus
Apple, Iphone, ipod, itunes, icloud
Hopefully with a new CEO MS will find a new identity for consumer tech and really get behind it
Bob,
Have you considered backblaze.com. — NO storage limits, and modest pricing.
Lets see your data is in California in an earthquake zone. Your backup is in Charleston in a hurricane zone. Hmmm.
.
I’d like to find a NAS/Could device like Western Digital’s My Cloud and have a way for it to seamlessly replicate its data to a cloud service, like Microsoft’s. It would be even nicer if I could keep a backup device somewhere else, perhaps at your house and it would be automatically replicated too.
.
I wonder what ever happened to products like Verity Topic. They could do a nice job organizing a lot of email into a database. Less space, easy to find stuff, etc. So many good ideas were crushed when Microsoft took over the PC software world…
You could try Unraid. I costs $ for their software, but you use your own disks. It has redundancy, and uses Reiserfs so you can recover data. I’m not associated with them and haven’t used it, but I’ve heard good things about it.
Seconded, it is neat.
For shame doing such a chintzy tactic just to get more space from OneDrive . What do you think? Microsoft is made of money?
Wait, they are. Never mind then…
For a while, I was doing similar tactics (but highly legitimate tactics. Honest!) to up my Dropbox storage from the initial 5Gb to around 15Gb.
However, I’ve since realized that if I really, really depend upon Dropbox, and I really, really like their service, I might as well be a paying customer. I signed up for their $100/year plan and I now have 100Gb of storage. After all, good companies producing good software is something we need to encourage.
For $25 per year, (a mere $2.08 per month) you could get 57Gb of OneDrive space and you’d have more space with less headaches since it’s all one account.
Besides, Microsoft could use the money. They’re certainly not making that on a lot of their other products. (Bing and Surface Pro, I’m looking at you.)
And if you need to move the data around while you’re juggling all of these deals there are some handy companies set up to make it easy… one of which is in my local community so I hope you don’t mind me mentioning them in a comment.. they have a cool domain name too: mover.io
Bob:
I just looked at Office Depot’s web site…..2TB (yes terabytes) for $200.
Why not buy a few of these, and keep multiple copies of EVERYTHING on multiple sites?
No “cloud”…..no NSA…..and YOU get to control all the backups…..
What am I missing?
“What am I missing?” You are trusting that the person responsible for each location won’t accidentally bump or unplug your stuff. If that person is you, then you are paying real estate taxes, utilities, an internet connection, and maintenance for each location.
I think Steve might object to you’re calling Spinrite “file recovery” software. It actually recovers and re-writes data (bits) on defective or deteriorating disks. But if the problem is that the file is corrupted or the file index is corrupted on a perfectly good disk, Spinrite will see that the individual data bits are all readable, so instead you would need “file recovery” software to undo the software corruption or restore deleted data. The point is that Spinrite doesn’t know or care about what a “file” is.
I take a lot of photographs with a high-quality camera and my need for storage keeps growing. I use HDR and bracketing and I’m slow to cull them. I was going to upgrade my NAS but after looking at my choices I decided to go with another computer. I’ve had enough of RAID and the different formatting on a NAS, so I have a 2×2 TB storage computer running Windows 7, a new one with a 4 TB storage drive running Windows 8. The photo archive section is the same on both computers. It is so much faster to find and work on the photos, the computer can do different things as well store photos depending on the software installed. Upgrading to a new computer is more expensive and more work than a new NAS,but I think it’s worth it.
If I send you all my emails could you save them for me? Or should I just use a simple password and let GCHQ automate the process?
Like you, I keep all my e-mail correspondence, but not in my inbox or, indeed, on my mail client. A few years back I wrote a mail archiver, tipped all my saved mail into it and then connected it to my mail system so it automatically keeps copies of all mail I’ve sent and received. This setup keeps archived mail in a Postgres database, currently holding 4.8 GB or north of 150,000 e-mails. Needless to say, I also wrote myself a search tool which can find mail to or from anybody within a data range (if I know it) and optionally containing a phrase or topic.
The benefit of using this type of mail storage is that searching it is much faster than ferreting through a set of mail folders to find an old e-mail. The worst case search takes about 25 seconds and normally takes under 10 seconds. The hardware is nothing special: just an old dual core Athlon box running Linux.
I’d recommend this type of archive to any individual, family or small business as a great time saver.
“I’d recommend this type of archive to any individual, family or small business as a great time saver.” So all they have to do is run Linux and write their own mail archive program. Simple 🙂
Monoprice just announced a NAS enclosure, RAID-capable (https://www.monoprice.com/Product?ab3=b&utm_expid=58369800-11.R-enhtUGRrSdHz5vzpVS2g.1&c_id=105&cp_id=10521&cs_id=1052110&p_id=11157&seq=1&format=2&utm_referrer=http%3A%2F%2Fwww.monoprice.com%2FSearch%2FIndex%3Fkeyword%3Dnas).
I wondered if it was worth looking into and how reliable it might be (cheaping out on such things can tend to bring heartache), but of course, it wouldn’t give you online, cloud-based storage.
Bob, Do you find the hard part of ‘book writing’ to be the actual writing, or all the peripheral activity (working with editors, wrestling with the software you type it on, getting involved in typesetting, etc) ?
Microsoft had to rename SkyDrive because Sky TV complained they had used their trademark.
Bob, is there anything that would stop you from doing this with Google Docs or iCloud? Or any other free service that only requires an email address?
I dont like Cloud for the same reason I dont like FB or any other online service offering. You lose complete control of your privacy and have to trust the other guy.
“Having a dedicated Spinrite machine makes a lot of sense because restoring a really corrupted disk that’s been sitting since 2005 or so can take up to a week of continuous churning.”
I agree about Spinrite. I wish Steve would offer a setting option that eliminated the home-seek recalibrate done between each read failure. Seeking the wrong track was a real problem way back when, especially on floppies (stepper motors vs servos) but not so much now. Plus, if you land on the wrong cylinder you can know that with a high percentage of certainty by reading several sector headers. If they are all the same =and= all wrong, then do the recalibrate.
That would take away most of that week, and save beating an already sick drive even more.
Steve is working on an update to Spinrite that drastically improves performance. It will be free to all Spinrite 6 owners. IMHO, he can’t release it soon enough, as it also should solve some problems with modern UEFI BIOS, AHCI SATA, etc. But, he’s currently distracted by a new project he’s calling SQRL.
I used to have a Seagate BlackArmor 440, but as mentioned above, the firmware is crap (standard web-based busybox retarded Linux) and barely maintained. I traded it out for a Western Digital DX4000 and have been quite happy, being a Windows-centric person (it runs Windows Storage Server 2008 R2). I would like a little more storage capacity than my current 16GB (12GB in RAID5), so I have been looking at alternatives. I spoke to QNAP the other day at SCALE12x and their rep mentioned that they have some models that will support Windows (those that run Intel architecture and have HDMI ports for video). QNAP of course has models with 6,8 and 10 disks or more.
And I have always used TrueCrypt, with which you can encrypt an entire disk or create a file that will operate as a virtual disk. And it’s FOSS.