In a few days we’ll be leaving Carolina, possibly forever. Following the recent death of my father-in-law — our reason for coming here in the first place — Silicon Valley calls once more. But before leaving town I was determined to scope out that $1 billion Apple data center in Maiden, NC. So I drove over, took some pictures, and talked to folks at the convenience store down the road. My conclusions from this unscientific research is that the giant Apple facility is mainly empty. It’s a huge building filled more or less with nothing and why Apple built it that way frankly escapes me. Maybe it’s just a shot across the bow of Google and its $650 million data center in South Carolina.
The place is certainly locked down. There’s a security gate on Startown Road and not much else to see. Apple has built a tall earthen berm around the entire site and planted trees atop that. The only way I could find to see the data center from ground level was from the Baptist cemetery next door. All that’s visible is the top of a huge white building and a couple of big tanks that may hold cooling water or, more likely, diesel fuel for emergency power generators.
I parked across from the main gate for an hour during the middle of the day and one pickup truck entered the facility.
It’s not that Apple has so little to do at the new data center which, after all, is supposed to be the center of iCloud and iTunes activity, updating all those Macs, iPhones, and iPads while serving video and audio to more than 200 million devices in all. That job isn’t trivial, but how much square footage does it actually take to do?
For comparison purposes, I looked at IBM’s Special Events Web Service (the gang that used to do the Olympics). They have 2000 square feet in three different data centers. They have a couple of racks of cache servers that handle over 95 percent of the actual work. Behind the cache servers are three racks containing about 50 1U linux servers setup in a cluster. These manage all the transaction work and anything that leaks through the cache servers.
Maybe the Olympics isn’t a good comparison, but where IBM has 2000 square feet, Apple has one million square feet — 500 times as much.
According to the Internet Movie Database there are about 700,000 movies in existence, excluding porn. Most movies will fit (in DVD form) in 4.7 gigabytes. Do the math and you get 3290 terabytes, which is a big number but not that big. Most data centers serving media files would cache about 10 percent for optimum performance. That’s 329 terabytes. Knowing a good percentage of movies aren’t worth the film they were printed on, you can probably come up with a 50 terabyte caching design and be able to serve anything anyone would want to see. Fifty terabytes of cache servers can fit into a couple of racks.
When planning a data center each rack requires about six square feet of floor space. But for the sake of discussion let’s make that 10 square feet to allow for non-server areas in the building. One million square feet divided by 10 square feet per rack means 100,000 racks could be constructed in the Apple facility. That’s 7.2 million 1U servers unless the racks are built extra-high, in which case there could be more than 7.2million servers.
Remember that between the 3290 terabytes of disk storage, 329 terabytes of cache and all associated servers, load balancers, etc. we’re talking at most 20 racks to serve every movie ever made. Now increase that by a factor of 10 because I probably blew a calculation somewhere. Now increase that by another factor of 10 because Apple may want to serve not only all our movies but all our TV shows, too. That brings us to 2,000 racks — two percent of the capacity of Apple’s data center.
Are you beginning to get my drift here?
Now Google isn’t Apple. Google is continually indexing the whole darned Internet, runs the biggest e-mail service, and many other services, not to mention all those ads. You can see how Google would require lots of data storage and servers to handle it. By now Google must have over a million servers and a very elegant way to manage them. But even Google’s million servers would require only 13.9 percent of Apple’s data center capacity.
So what is Apple doing with such a big building? I can’t imagine a workload that would need even a tenth of that data center.
Maybe they are building for the future, you say.
That’s crazy. Remember Moore’s Law? As time passes all of those Apple racks will be filled with new computers that are faster and have more memory and with storage systems that hold more, too. The square footage requirements are, in fact, likely to stay about the same for the foreseeable future, absent some quantum expansion of Apple’s services.
For that matter, what servers is Apple using, anyway? Certainly not the now discontinued xServes. That alone may be the reason why they’ve made the facility so difficult to see, not wanting to boost any competitor by admitting Apple is a customer.
So here’s my guess: I think it’s a joke. The building is a near-empty facility built primarily to intimidate Apple competitors. And so far it seems to be working.
1. Based on the aerial photos I’ve seen I’d bet those tanks are for water. Diesel fuel is usually stored underneath the gensets themselves in large belly-tanks.
2. FYI The rule of thumb for building datacenter is 25—30 square feet per-rack. That allows for the rack itself (which have grown quite DEEP** over the years, as server specs only call out height and width, leaving depth to expand almost infinitely!) Your initial figure is a very old one, based on now-obsolete 2-post racks from the telco CO days.
http://serverspecs.blogs.techtarget.com/2007/09/17/a-myth-busted-1u-servers-do-not-provide-greater-density/
Otherwise your article is spot-on. The scale of this facility is mind-boggling.
The tanks are definitely for water to backup the chillers.
I’m sorry for your loss Bob.
As for your conclusion, well, that seems crazy to me. Why go to such lengths? what purpose would it serve? Is Google going to be intimidated?
By the way, SUCK IT Robert M. Young Jr! (or whatever your name is)
I’m sorry for your loss of composure. I didn’t mean to hurt your feelings. Did I call you a naughty name????
Kidding. I was just happy to post before you did for once.
Has anyone checked out the size of their electrical service? It sometimes can be tough to look from afar at a house or small commercial building and tell the difference between a 200A service and a bigger one, but at the scale this facility could be operating at, you might be able to tell from an aerial photo. (Bob, did you get a photo?) We can’t know how much they are drawing over that service, but do the service equipment and connections match up with the potential number of racks that are being discussed?
Regardless, the low vehicle traffic seems telling…
Apple’s data appeared on Google Earth about a month or so ago. Zoom in on 35 deg 35.290′ N and 81 deg 15.699′ W.
To the left (west) of the building you can see the power utility substation. I am not practiced at recognizing high voltage systems — it appears to me they have only 1/6th of the planned transformers. Someone can probably guess how many KW is being supplied to the site.
It looks like they have a single utility feed to the site — interesting. I don’t see many stacks for the generators. I wonder how many they have installed.
The big tanks are near the cooling towers. So they must be for water storage. I am not an HVAC engineer — it appears to me there is not a lot of water storage, probably enough to run the site for a few days.
It looks like they have a water pond and a couple pumps set up for a backup fire water system.
I find the loading dock areas on the NE and SW sides of the building interesting. There is not enough room for a normal sized tractor trailer “semi.” This is going to make the delivery of large numbers of servers interesting.
If anyone wants to see the facility on plain-old Google Maps, you have to enter the coordinates in this format:
+35° 35′ 17.4″, -81° 15′ 41.94″
How many transformers are running is a question, but that substation certainly gives them the capacity to be running a lot of equipment at that site! No good clue there.
My sense is that the loading docks ARE suited to semis – just not many of them, nor set up for rapid turnover of trucks – so not good for manufacturing, but good enough for delivering server racks periodically as the site scales up.
I don’t know anything about the soils in the area, but the ponds may primarily be for detention of storm water runoff. All that roof and roadway means a lot of impermeable surface, thus the need to detain storm water. It’s interesting that the roof has no “green” component. The current modular systems with engineered soils and low maintenance plants make extensively planted “green” roofs very easy to build and maintain on new buildings.
I measure (using Google Earth’s ruler) the space between the loading dock to the wall to be about 82 feet. A typical semi-truck/trailers run between 55 to 70 feet long. The minimum recommended apron length is 135 ft. NO a semi-truck/trailer won’t fit. I just did the math, there is not enough room even if you dock the trailers at a 45 degree angle to the building.
In addition to the docking areas along much of the full length of either side of the building there are additional docks on either side of the smaller concourse at the front of the building. (There is a third dock at the front that is certainly constrained by the small “higher traffic” car parking lot.) These are designated as J and K in the property assessment here: https://www.gis.catawba.nc.us/nomap/AssessmentReportCom.asp?key=362711760090&card=1 There is much more than enough clearance at either of these docks. At this point, K is likely capable of fulfilling all future expansion, repair, replacement needs excepting extreme emergencies.
Who justifies web comments? Cringely really is off his rocker. (Sorry about your family loss though.)
I should add that the areas along the length of the sides of the building (the area I believe you are referring to), although I believe there are docking bays for servicing, are walled in and appear to be primarily for ops/maintenance/mulit-use purposes rather than receiving.
From https://www.charlotteobserver.com/2010/10/05/1741621/apple-pays-maiden-couple-17-million.html :
It’s 500,000 square feet, not a million. For comparison, Google’s installation in The Dalles occupies 200,000 square feet and the Facebook data center in Prineville will build out to 300,000 square feet. There’s nothing outlandish about the size of Apple’s plant.
It’s an automated plant with 50 employees. If you want to see them, park by the gate at shift change, not at lunch time.
The problem with the rack space calculations here is the implicit assumption that each movie is streamed by only one user.
This is the most amusing post I have read in a long, long time!
Maybe it’s room to start their own chip fab…
When talking about the new spaceship that Apple plans to build in cupertino, they mentioned that power generation was going to be *in* the building, and that they’d use the standard power grid as a backup for the building’s own power generation.
My guess is that they’re doing something similar for the datacenter in NC. So some of that building area might be for power-generation. It’s still huge, of course, but that might be where *some* of the space is going.
As one other datapoint, I know that when we announced plans to build a super-computing center in upstate NY, one of the first calls we got was from the power company. If Apple does have a huge number of servers, then they will need a serious amount of power to run them.
Data centers are mostly unmanned. Even Terremark’s 765k sq. ft. center in Miami FL rarely has more than a handful of people on the floor – there are more guards than techs usually.
Most data centers seem to focus on 2U – storage servers are generally 3U-4U+. Blade servers usually are 7U-10U, but, in terms of capex, Blade servers generally don’t have the ROI needed for a large data center and are not deployed as frequently. Being Apple though, I don’t know if they are using 1U xserves + some SAN/NAS.
Cooling is a large portion of the building – probably taking 10%-15% of the internal space – possibly on the exterior walls of the building.
As it is probably cloud support, it is possible they have two or three redundant copies of that data.
Data centers have PDU rooms, battery/flywheel rooms, fiber/telco meetme rooms, etc – all of these take up space. Coupled with OSHA compliant walkways, proper hot/cold aisles, etc, you lose a lot of floor space.
You’re also assuming that it is 100% filled out. They could be looking at growth, the time it takes to build a data center versus just trucking in more equipment. The incremental cost of a million square foot building over a 500k squarefoot building may be less than the cost to build a second 500k sq ft center not including the time required to get permitting, power, etc to build a second data center.
As for the number of people coming in and out, if their network anticipates failures, failed machines can be pulled off the rack and replaced at any time – any single failure (or multiple machine failures) probably isn’t enough to impact anything and can be fixed on an as-time-permits schedule – meaning their 24×7 staff can be quite small. During the day, they may have techs that grab a pick list and pull broken machines and replace them with working machines.
You forget that Apple in the future will automatically save everything you create on the Mac or iPad to the cloud. Every single file.
Because of the horribly expensive infrastructure that goes into any top-class DC you always build big, because somehow you find a use for the capacity. Datacenters come in two varieties, space- and power-limited. Space-limited DCs are only found in mature development areas – downtown Seattle and NYC both have commercial DCs built into multi-use skyscrapers. (This caused problems when the Westin hotel in Seattle caught fire and the fire marshal refused to allow the DC personnel to remain in the building, causing several large websites to go dark.) Because you can only estimate the size and power needs of the next generations of servers and other DC equipment, you build big. I’ve worked in a DC in Denver that had a room that could have hosted a football game, complete with bleachers on both sides, with room to spare for a basketball court on one end. We used about .05% of that room to store spare racks and other random stuff. Most DC’s have empty space like this; once the power is fully committed, the excess space gets parceled out for other uses – workspace for clients, storage for spares, a weight room for the staff. The clients simply don’t know about it. For more check out James Hamilton’s blog, Perspectives (http://blog.mvdirona.com) – he’s the DC guy for Amazon.
Also consider that they’re probably moving all iTunes (music) and iTunes apps to the same location. And they probably have some levels of data redundancy. They’re not going to run an operation this important on a *single* copy of every movie they sell.
That’s probably one of the reasons that music-matching is in *their* interest. Would you want to guarantee the storage of a million songs [including data-redundancy, etc], or a million slightly-different rips of a million songs?
Redundancy came to my mind when reading this too. I wonder if they’ve given any thought to geodiversity.
I’m no Apple fan, but as successful as they’ve been, it’s difficult to picture them making a basic mistake like building a data center that’s orders of magnitude bigger than it needs to be, or putting all their data in one (hurricane prone) region.
And I agree with the other poster that Apple doesn’t need to try intimidating anyone. Besides, none of the sheeple will use Google or Amazon services unless Steve Jobs gives the OK, which seems highly unlikely to me. iPod fans have been ignoring Amazon and overpaying for music for years, and I don’t see any reason to think that’ll change anytime soon.
Yeah, sheeple, whatever, pseudo-individualist.
Quality products with an excellent philosophy of business and tech is what makes Apple what it is and what makes many support the company – shiney iPods only make it immensely profitable.
You should learn to appreciate the difference.
My condolences to you and your family on their loss.
Probably related to a combination of tax incentives, future uncertainty, need for flexibility, relative cost of over-building vs under-building and the simple fact that Apple is hyper-confident of its 10 year horizon.
google maps has a picture that looks like it was taken mid to late afternoon. there are maybe 4 cars in employee parking spaces and a pickup truck in the loading dock.
My condolences as well.
For what it’s worth, I read recently that Netflix needs to maintain many copies of each movie/item in their library, since all the different streaming-capable viewing devices (PC, Roku, game box, TV, Blu-ray, etc.) expect different digital formats. They have leveraged Amazon S3 rather than build out their own data center. Apple’s ambitions are much grander of course.
Hey… who is Robert M. Young Jr. / whoever and why is he supposed to suck it?
I missed something along the way…
You missed a too-obscure in-joke, I think.
I think Robert Young is either the Red Hat guy (but he’s usually “Bob”) or the database guy.
It’s got to be the database guy in this case, and I conclude that because anyone who says “you know, sometimes using the relational model for data is actually a good idea” sometimes gets attacked by elements of the “NoSQL” crowd. And whenever you’re dealing with the sort of volume of data implied by the size of this data center, the NoSQL crowd picks up the scent of their natural prey.
That’s all I think was going on there.
Yes, it’s an in-joke. Robert Young is usually the first the respond to an article and I just happened to beat him to the punch today.
No offense meant!
Alas, my First record here isn’t so hot; I’ll go days without checking in. And this thread is boring, so I wasn’t composing an on-point response. Bob’s just fitting himself with a new tin foil hat. Some database specific sites, my record is a bit better.
Secondary space ship launch site.
They are going to get to the moon ahead of you Bob!
They will need some of the extra space for prisoners, like the infidels who have said mean things about Steve Jobs ;-), asked for a refund on an App or something equally terrible.
Finally, and this will take up the bulk of the space. Tens of millions of Apple followers who have, up until now, had their heads in the metaphorical clouds, will probably want to put them in the new, virtual, iCloud.
But I could be wrong.
They’re baiting press coverage.
Sometimes Steve Jobs just walks into the near-empty building and hangs out. He finds it relaxing.
Semi-off topic, but more interesting to me than what vendor provided those servers is whether or not they’re running Mac OS server or another operating system.
From Jobs presentation we got a glimpse and they certainly aren’t Sun gear which rules out Solaris. Interestingly they could be HP or Dell boxes or home brewed like Google’s but I’m betting if Apple were building their own servers for this operation, they’d have kept the Xserves going even if it meant they were their own biggest customer for rack servers.
That leaves a vendor like HP. You know they’re not running Windows, so that really leaves Mac OS, BSD or Linux. What’s really interesting is if Apple got HP or Dell to add support to their custom firmware for Mac OS. However, that’s not likely so I’m guessing that rules out Mac OS. Since HP and Dell have solid Linux support, I’m going to go with Linux.
Arguably, Apple may have discontinued the xServe just to *be* their own biggest customer.
Just a thought.
I have no inside information on what Apple is doing with their data center, but I think spending a billion dollars on a bluff is extremely unlikely.
Bob’s back of the envelope calculations about movie streaming are right, if all Apple hopes to do from this facility is stream pre-existing video, but I think there are really two questions here: 1) why would Apple build this huge data center and 2) what will it do with all that capacity?
The first is, I think, the easier one. The quote attributed to Thomas Watson about there being a market for five computers in the world is coming true, in a way. There are a very limited number of top tier Internet infrastructures: Google, Amazon, Yahoo (don’t count them out just yet) and arguably one or two more. We know Apple has a tremendous aversion to anyone else being on the critical path for any of their platforms. Since mega-scale Internet infrastructure is becoming a critital platform, it’s clear that Apple would want to own one.
Now that they have the capacity to be a top 5 cloud player, what will they do with it. Streaming movies is just capacity bits; the Apple we know would want to do something more. From what Apple has published about Photo Stream in iOS 5, it sounds like they will already get nightly backups of your photos and videos shot on iOS devices. What if that ties into an Apple supercomputing cluster for realtime editing and rendering? From your iPad, the video editing app just sends a list of cut points and transition cues to the data center which already has the source material. 1080p rendering (and output to your Apple TV) is done in real-time. What about 3D rendering for animation? With a million square feet of data center space, they could render thousands, maybe hundreds of thousands, of Pixar-quality render streams in real time.
I agree with the previous commenters who talk about data centers as being either power contrained or space constrained and will add cooling to the list. Comparing land and power costs in North Carolina to the same in Santa Clara Country, there is no reason for Apple not to go huge.
They are going to house Steve’s brain in a vat and have it run the world from there. The immense building is necessary for life support with today’s technology, but as the decades go by the required footprint will go down. Eventually the required equipment will be reduced to the size of a small backpack, yes about the size of the first Macintosh. Steve’s visage will appear on as a lifelike 3D projection above, and say “My name is Steve Jobs, and . . . . (remainder left as exercise for reader)”
I’ve got to wonder if they might not be using some bizarre combination of, essentially, AppleTV and “Time Capsule” systems.
They’re getting more and more experience with running the Darwin kernel on ARM processors (“it’s not just for handhelds anymore”), and they’ve got their own custom ARM processors. I wonder how the “compute power per watt” and “compute power per square foot” calculations add up if they do something custom and ARM-based with storage off in a SAN.
I’d kinda love to see the amount of power the facility draws, and the amount of heat it dumps. Quick, someone go visit at night with a thermal imaging system!
Now, insane conspiracy fodder: hm… given Moore’s Law and Watson and Wolfram Alpha and the calculated computational complexity of small rodents and stuff, based on projections, how large would a data center have to be if in ten years we wanted it to match the theoretical computational complexity of, say, a human being?
If a rack uses a LOT of power, it is common to put more air space around it. This lowers your watts/sqft value and keeps the HVAC guys happy. Of course the way data centers do climate control is so primitive its crazy. Visit a 25 year old auto assembly plant and you’ll see much, much better ways to do this.
Fuel storage — there’s another thing data centers do differently and is crazy too. Real facilities have a storage facility that can be easily serviced by tank trucks. You keep a few weeks of fuel on site and have a contract to get more in an emergency. Funny things happen to diesel if you let it sit too long. A good facility will have equipment to keep the diesel fresh. Near the generators you put in “day tanks.” These tanks have just enough fuel to run the generators for a few hours. When the generators are running, you replenish the day tanks from the fuel farm. Day tanks are usually positioned higher than the generators so gravity can get the fuel to the engines. Gravity is very reliable. Sometimes things happen to diesel engines. It is best not to have too much fuel near them.
Belly tanks right under the equipment… If you have ever seen a diesel fire you can appreciate how scary this approach really is.
These are perfectly good 50 year old design techniques. Long forgotten in the data center and nuclear power world.
so – the water towers are for cooling their own nuke plant?
What if the extra space is to house a ton of Bloom Boxes to make the power requirements greener. Then sell the excess power to the grid?
Sorry about the loss Bob, great article and will bring a ton of speculation out.
My company is also building a data center nearby in Kings Mountain, NC. One of our largest at 200,000 square feet, as pointed out by others a good portion of this space will be taken up by the battery room, and the UPS room and the generator room. Add to that a caged area for shipping/receiving/storage. Add to that a network operations center. The security lobby/mantrap area. Break room, restrooms/showers/fitness room. Office space for support staff located at the facility or transients. Conference rooms/telepresence.
All of a sudden your 500,000 spuare foot data center building has ~350,000 usable raised floor space.
Now you need outside power. For such a large building and potential usage, you have a very large power envelope. Your local power company is probably going to need to upgrade your transformer. Maybe you want them to site a redundant transformer. Project time: 12-18 months. Even for Apple.
Then you need multiply redundant fiber. 6-12 months planning and construction. Sure, this can be going on concurrently, but everything takes far more time than you would think.
As for your raised floor (which, during the .com boom, raised floor panels were frequently backordered due to supply constraints) more than likely these days you’re cutting that up into smaller, more efficient rooms that you can build out as necessary, so you’re not cooling 350,000 square feet of raised floor when you have 20,000 square feet of cabinets. You might have 5 or six rooms.
The one thing Apple doesn’t really have, as you pointed out, Bob, is a server. So what are they using? Are they using off-the-shelf hardware from another vendor, or are they rolling their own under the radar? The most likely is that Apple is using some combination of off-the-shelf x64 hardware that would run some form of MacOS (ServerOS Core, anyone?) or UNIX/Linux. I’d say it’s more likely they would be running some Sun hardware for things like Oracle databases to keep track of all the media. It could be that they’re using some of the new Cisco UCS blade chassis, as they seem to be curiously constrained right now.
They could have something up their sleeves, however. I don’t see Apple porting MacOS to SPARC, and I don’t see them using HP Superdome’s, either. Google has a couple of suppliers make custom hardware. I wouldn’t put it past Apple to do the same. Maybe not a resale product, but definitely something built to their exacting specifications. They way these projects go, and the way Apple likes to be green, perhaps they’ll delve into submersible cabinets.
Whatever they will be using, they will need lots of storage, far beyond 329 terabytes if they intend to provide every iCloud user with 25GB or more a la SkyDrive. Yes, they will be able to take advantage of Dedupe, but that only goes so far. So NetApp or EMC (or Oracle) will be shipping some racks of disk shelves as well.
I just don’t see spending $20 million or more and letting the crickets take over. I know WE’RE not. But there are a lot of other requirements that need to be met before you start rolling in cabinets.
First off, Bob, my condolences as well to your family. I imagine there have been some tear-filled days recently, and I hope the ongoing healing process will be fruitful for all of you.
Secondly, I am sorry to say I am deeply disappointed in something from your post. You have lived in Charleston, SC for how many years now, and yet you still wrote that in a few days you would be “leaving Carolina, possibly forever.” You mean you’ll be leaving *South Carolina*, possibly forever. We North Carolinians are a different bunch from the folks farther south. This is not California, with a “northern” and a “southern”. We are two distinct states, with (at least) two very different personalities.
Did you ever make it up here to RTP while living in SC?
Have a great trip, Cringely family.
Everyone talks about how Apple does not have a server because they no longer sell the xserve. Just because they don’t sell the xserve does not mean they can no longer produce the xserve or a variant.
Hm… Based on some napkin calculations that I won’t bother to repeat here as they’re wildly approximate and make some assumptions that I’m probably completely wrong about, a million square feet of data center won’t be sufficient to store an accurate atomic-level record of the structure of a human brain for another 40-odd years (assuming Moore’s law holds). So, we can safely rule this out as Steve Jobs planning to become the first fully uploaded human consciousness…
I think there only reasonable answer for Apple to build such a gargantuan server facility. Steve Jobs has been contacted by aliens who need to “educate” us. The capacity is required due to the need to store all the relevant knowledge. It’s to bring us up to speed before they’ll waste their time on us.
I watched the film of Jobs engaged in Q&A with developers at WWDC 1997. He said Apple would be focused on enabling their clients to “live” on the Net, having their computers backed up and their computing environment available on their computer devices anywhere. It’s taken 14 years so far. N. Carolina’s facility may be just another step along that path. It’s early days.
My figures indicate that is the size required for the until now unknown Apple MacIntosh data cluster.
Clearly Apple is gearing up for something; something very, very big. Something spectacular. A portal? Movies, music, documents, time capsule, photos?
Apple: the final frontier. This is the new location of the starship Apple. It mission: to explore strange new worlds, to seek out new life and new civilizations, to boldly go where no one has gone before.
You’re traveling through another dimension — a dimension not only of sight and sound but of mind. A journey into a wondrous land whose boundaries are that of imagination. That’s a signpost up ahead: your next stop: the Apple Zone!
You unlock this door with the key of imagination. Beyond it is another dimension: a dimension of sound, a dimension of sight, a dimension of mind, a dimension of cloud. You’re moving into a land of both shadow and substance, of things and ideas. You’ve just crossed over into… the Apple Zone.
There is a fifth dimension beyond that which is known to man. It is a dimension as vast as space and as timeless as infinity. It is the middle ground between light and shadow, between science and superstition, and it lies between the pit of man’s fears and the summit of his knowledge. This is the dimension of imagination. It is an area which we call “The Apple Zone”.
I have no idea what they need all that space for now, but I guarantee that in 20 years they’ll be thinking they should have built it bigger…
My company just built a 260,000 sq ft data centre. 30,000 sq ft is available for rack space. The rest is cooling and power, with 15,000 sq ft for office space and sleeping quarters etc. Not a lot of rack space for the thing’s size. It’s rated as Tier 4.
I’m amazed that not only Bob missed it, but every commenter so far: every Apple device owner (there are 200 million iOS devices out there according to Apple plus at least, what, 30 million Macs capable of running Lion) will be entitled to 5GB worth of iCloud storage for their mail, photostream, unmatched by iTunes music, documents and probably future unmatched by iTunes movies & TV shows once those contracts get signed. Obviously people with greater data requirements (like those old MobileMe customers like me used to 20GBs of space a pop) may want to pay Apple for extra storage. And on top of those 230 million existing devices, Apple will conservatively sell another 75 million this year and 160 million next year.
So instead of multiplying 5GB times 700,000 as you are doing, you need to multiply it by 500 million.
And that is presuming Apple never comes out with another cloud based, data hungry service, ever again.
I suppose you are correct. Five gigs times 250 million devices is a lot of storage even if most would fill only a third of it, which was my experience building a somewhat similar capability minus a couple zeroes many years ago. Though it’s less than the storage currently promised by Google to its Gmail users from what I understand is less data center space.
There’s another (though similar) reason why they might need so much space: Apple just concluded a deal with the record companies allowing iTunes to match identifiable songs in user libraries. Imagine a record company negotiation where Apple explained that its new data center could handle storing every user music library on a 1-to-1 song basis — a TRUE music locker. If Apple was willing and capable fo doing that, then the record companies would lose their leverage of demanding it and would opt for an extra point or two of margin. But you have to own the big gun if you are going to threaten to use it.
So Apple were rumoured to have purchased a 10Petabyte EMC/Isilon array for their itunes / cloud storage needs. EMC now can do close to 1PB per floor tile. So say they conservatively cram their new 10Pb array into 20 racks, and through de-duplication the array can store 10 times that (100Petabytes) of end user data, what the hell is the rest of the space for……
Welcome back to Cali. My condolences for what prompted such a trip.
Your article this week is rather curious. But I think your conclusion is spot on. Intimidating indeed!
You really have fallen off the deep end, haven’t you?
You draw these conclusions from talking to convenience store clerks nearby? That moonshine you were sipping with ’em must have been spiked with methanol! Because you’re blind at a bat!
Moore’s law has nothing to do with data centers. And didn’t you see the photos of the interior of the data center? Have you ever even been in a data center? I suspect not.
Pathetic writing.
Some other ideas (though it doesn’t account for all of the space):
What if Apple is trying to corner the education market? Textbooks are queuing up for the iPad. Online courses and instruction are sure to follow. Apple.edu?
The medical records market? Google just got out of this arena. I just viewed my daughter’s x-rays on her iPad … Apple.rx?
Square? Apple.money?
The possibilities are endless. Amazon has great cloud based products if you have the required skills to leverage them. Apple will make this seamless. No rtfm required.
Like it or not, everything has changed with Apple taking the Cloud seriously.
I’m probably wrong but from the photo that was shown at the WWDC Keynote it looks like Apple is using SuperMicro hardware. I’m pretty sure MobileMe ran on OpenSolaris so it would not surprise me if iCloud also runs on it also.
As to the estimates of floor space for the server racks it’s not just the square footage of the rack itself; you need lots of room for humans to walk around the racks and have plenty of space to pull those servers out for servicing. So, if the rack is two feet wide and holds 39″ deep servers you have about eight square feet right there. Now you need to multiply this by at least four to allow room to service the front and back of the rack. You’re looking at 32 square feet per rack.
A data center of this size would need redundant power and redundant cooling systems. While cooling systems are usually roof mounted the air handling equipment is going to eat some floor space.
The battery systems eat loads of space as well.
A comparison to Google is also not exactly fair. The systems doing the work of providing search results are not running state of the art hardware. A premium is placed on energy efficiency and reduced heat output.
And Google rolls in a rack of servers at a time. They don’t even bother with servicing or replacing components. They just turn it off and leave it in the rack. The servers are all open top, proprietary design sleds.
Are they using Mac Pro servers?
Here is a view of the Apple data center from a helicopter:
https://www.youtube.com/watch?v=hDXSSi1qStA
[…] data center as his family traveled from the Carolinas back west. From his brief, midday sojourn, Cringely concluded the following: [T]he giant Apple facility is mainly empty. It’s a huge building filled more or less with […]
“According to the Internet Movie Database there are about 700,000 movies in existence, excluding porn.”
Please don’t exclude the porn.
1. This is actually just a landing pad for the flying saucer they’re building on the west coast.
2. This is a church. The faithful will come to live here.
3. It’s where they’ll house Netflix when they buy them.
3a. It’s currently mostly sublet to a small company called Skynet LLC.
4. It’s also a manufacturing facility.
4a. The flying saucer will be constructed here, propped on end and rolled to California.
4b. All future devices will be built here by imported Foxconn employees housed/stored/imprisoned in the facility. The issue with the loading dock not being big enough is a big mistake.
4c. Skynet LLC will manufacture robots of some kind here.
4d. The iCar will be manufactured here.
5. The reason it appeared so quickly on Google is because it is just a shell to hide the excavation going on inside.
5a. The guy in the pickup truck had dirt in his pant legs.
Apple just needed a building large enough to hold all their cash.
There’s detailed info at https://www.datacenterknowledge.com/the-apple-data-center-faq/
[…] data center as his family traveled from the Carolinas back west. From his brief, midday sojourn, Cringely concluded the following: [T]he giant Apple facility is mainly empty. It’s a huge building filled more or less with […]
There are lots of data points out there that Cringely is not taking into account;
offering 5Gig storage for every iOS and OSX account 500 to 1000 million gig of simple storage and
offering back up of versions for 30 days put a multiple on that of 1.5 to 3.
Add optional paid higher storage options(already announced that like MobileMe you will be able to pay for larger cloud storage would guess that less than 10% will opt for this but those who do will use something in the range of 200+ gigs (that is what my family plan expanded sore on mobileMe is now.
All of the above is just for storage arrays (plus back-up and redundancy.
And this is before you get into Music content – As of February 5th, 2010, Apple reported the App store as having 12 million songs. This announcement was in connection with their announcing their 10 billionth download.
Source: http://wiki.answers.com/Q/How_many_songs_does_the_iTunes_store_have#ixzz1QeayIQRe
So 12 million at at multiple bit-rate sizes, all of the video – theatrical movies in three of four formats, All the TV shows, all the Pod casts (most in 4+ formats), all the ibooks etc. And again this is just basic storage.
Front end service for 100 million connections will require massive amounts of processor and network gear to handle the requests with low latency response, and massive caches to stream out multiple copies of most of the static content to millions of users requests.
Add in some cool stuff clearly on the horizon: Apple location based services, iAds, server side voice recognition, along with all the Apps for OSX and iOS app stores, plus data for tracking every transaction.
And while Google has lost of web mail clients, apple is offering a free account with every iOS5 and OSX10.7 system for life. That could easily grow to 200 million over the next 24 months.
All these services require storage arrays, local cache storage and lost of processors to handle all the different requests, data base searches and transactional support.
I am sure there is big growth in the building and they have the basic permits for doubling the size of the facility in place.
I have worked in a few data centers in the Seattle Area and the original MSFT hotmail data center that turned out to be too small not offering enough redundancy and provided very limited size data per email as compared to todays systems still occupied more than 50,000 ft2 and it had only 2 techs as regular staff and some guards.
It did get busy during routine failures with guys running around trying to figure what was going on, but that was when load balancing / data center management software was very primitive.
Apple already has a 109,000 square foot Newark, Calif. data center Apple bought in 2006 and this has proven to be marginal or worse during their current peak traffic loads for updates, and mobilMe traffic.
Datacenters don’t need to be crowded or busy places with lots of people going here and there all the time.
I’ve been working for 3G telcos and I can assure you that I’ve found no more than 2 persons in a datacenter at any usual time.
Big things happen during the night, aka “off peak” time slots.
Bob, do you want to know whether that building is a fake datacenter or not?
No way without actually entering it and playing around. But …
1. Real datacenters eat a lot of energy. So the electricity meters need to run high, but I have no idea on how to check this. But there needs also to be a huge amount of heat coming out of that building. This is easier to check with an IR visor (go to your local spy store for discount prices), especially in the night.
2. Real datacenters need to be connected. So there must be a number of data links. 99% of which will be wired links with poles, cables and manholes (for buried cables) visible. If the links are wireless (which I think is not reliable for big-A) then your favorite magnetic fields detector (again, your local spy store has plenty) will show interesting activity around some specific GHz frequencies.
3. Real datacenter have people coming in and out from time to time. So a single sneak won’t show significant details.
Honestly, I’d say an empty huge building a very expensive deterrent, even for Apple.
But I can easily be wrong, as usual.
Microsoft has finally release its online version of Office. I think it’s called Office 180.
you are correct; Apple has no need for such a data center. It could be a black-budget military site with Apple’s logo on it. Not sure if Steve Jobs would be OK with that but bureaucrats in the military always have leverage.
Peter
If this is a datacenter…
Apple could be using the facility as leverage, a threat to onshore some of it’s manufacturing.
Apple’s Service Center program has been down for 6 months ( https://www.apple.com/support/programs/aasp/ ) they could be gearing up to bring all of their service in house. Proprietary iMac HD connectors anyone?
It could just be storage/distribution center.
It could be there is a zoning or public utility regulation there that equates sq ft to power draw and if they had build a smaller facility they would not have been able to ask for as much power as they plan on using.
It could be that Apple asked for estimates an was told this was the cheapest size building to build (don’t laugh I’ve heard such lines before).
It could be so many things but it is probably to early to tell much of anything. This is after all Apple and they try to be the masters of “just one more thing…”
Well any datacenter I’ve been in requires a lot more than 6 sq ft per rack. That is the footprint of the rack at 24″x36″, but the racks are spaced by minimum 36″ and more typically 60″. Then there are the hallways running the other direction. So I would put the estimate at around 20 sq ft per rack.
I have also heard many times that the datacenter is 500,000 sq ft, not 1M. So that puts it at 25,000 racks (still a huge number, I agree).
For the number of servers they may not all be 1U servers, plus the racks might have switches, power conditioners, maintenance trays, who knows. I’d call it 50 servers per rack being generous and 25 being stingy. So 625,000 to 1,350,000 servers.
Now consider that it is sized for the existing 200 Million iOS users, 60 Million Mac users, and the 120 Million iOS+Mac users that will buy stuff in 2012, and the 180 Million iOS+Mac users that will buy in 2013, the 240 M in 2014, etc. and it starts to make sense. Call the number of supported users 200M to 1B.
All those users get email. All get app state storage. All get doc storage. All get photostream storage. All get 5G or general storage.
So it may be underutilized now, but it is built for massive growth.
Bob, you are a pilot. Rent a 172 from the local club and fly over. Not current? Get that pimply-faced instructor to go with.
Knowing what we know about Steve, you have to think Massive World-Conquering for any of his plans. What would put the Mac and Apple at the very top of the computing world heap? He wants the iTunes store to sell EVERYTHING! Every movie, every TV show, every Book, every computer program, every thing that can be digitized and sold in bits and bytes. He wants world domination for Apple.
Two words: Vacuum tubes.
Jobs even said in the WWDC keynote that this is just one of 3 other data centers. It’s likely still not fully complete yet. It’s been stated there is another 500,000 square foot building going up next door to the first. Who is to say that servers have even been installed in the NC facility? Without power and data stats the building is likely not even turned up all the way. Forget about just the USA customers, the cloud is going to have to scale worldwide. They may sell additional storage over the free 5GB’s. There may be entire plans yet to be announced. Apple always plays 3-4 steps ahead of what is public knowledge. They make deliberate moves one or two at a time until suddenly they announce, “Check and Mate”. Judging from their outstanding sales, I suspect there will be insane growth rates and the data center is in place ready to scale up iCloud quickly as demand jumps.
Elementary my dear Watson: it simply cannot have been built with commercially produced material alone in mind. Movies, music, apps, etc., have easily estimated storage requirements, especially given the de-duplication that iCloud will be doing on all our iTunes collections. So what’s the elephant in the room?
Personal data. Personal data squared, cubed and more. Exploding numbers of Apple device users, producing photos and videos of exponentially rising number and resolution, documents of rising number and complexity (n.b. the long-postponed paperless world). Factor in new, high-bandwidth services (e.g. iChat video answer machine – http://forums.appleinsider.com/showthread.php?t=97189 ), perhaps services open to all computer users regardless of device, and a future factor to push back the possibility of capacity issues for a decade or more, and it all begins to seem less incredible. The size seems incredible, but ten years from now, perhaps not so much.
As Alan Kay said, the best way to predict the future is to invent it…
Maybe it includes a cryo facility for Jobs. He looks worryingly ill, and shareholders may have demanded a way to keep him on ice for when the company needs rescuing again. What better place then where you need cooling anyway? Could even be the precursor to an iMausoleum or an iTheme park..
Joking aside – no data means no data. As long as Apple doesn’t say what it is it could be anything, an iSpace Shuttle development facility, an iPresident – you name it. It would be fun to see just how mad you could speculate before Apple brand management decides it has enough..
Oh, and one other thing: if this *really* was a data centre it needs at least a twin somewhere else for redundancy. If that doesn’t exist it’s unlikely to be a data centre.
It’s one of three datacenters.
Its obvious.
They need that much space to store Steve Job’s ego once digitized.
LOL. Good one.
You left out the day care center for the kids of the technicians.
iPorn?
Here is a good look at Apple’s data center.
As a sport pilot myself, wonder how long we will be this free to roam.
http://www.youtube.com/watch?v=hDXSSi1qStA
It’s obvious isn’t it – there’s no need for any expensive cooling since all the servers are so far from away each other 😉
High resolution audio and video?
Digitized health records?
iTube?
If I remember correctly one feature of the iCloud was that it would also hold all your own music and videos too. If that’s the case it’s a much bigger task than just storing every film ever produced. So Bob’s calculations go out the window.
Quick example. If I’ve got Triumph of the Nerds to watch on my iPhone and sync it to the iCould that’s one copy. But Billy Nomates does the same, that’s two copies of the program. And so on. Even though it’s the same program, we each need our own little bit of storage.
That’s how the space requirement will easily multiply.
Don’t cloud storage systems generally detect and eliminate bit-perfect duplicates to mitigate this issue?
That’s a good point, but if I do my own conversion of Triumph of the Nerds, and Billy Nomates does his on different program, then we both end up with different varients.
Unless they have a piece of cleverness that examines your version of “Louie Louie” confirms it as a duplicate of everyone else’s and only keeps one copy linking your iTunes library to it.
Looking at the fly-over and readers comments over at Techcruch from last July
http://techcrunch.com/2010/07/20/apple-data-center/
Some observant reader pointed out the 13 truck loading docks.
Sounds like a warehouse or production facility with a data center on the side.
I am sitting in a building with about 50,000 square feet of data center space and it has 5 loading docks. It has about 20,000 feet of office space too. The Apple facility is 500k ft^2 with about 184k ft^2 of data center space (see below for more stats). The property assessment may be a better source then a guy who took a picture of the gate and spoke to some folks at the Kwiki–mart.
The stats from https://www.datacenterknowledge.com/the-apple-data-center-faq/ :
“The Apple data center in Maiden features a one-story data center floor spanning 184,814 of space, according to a property assessment from Catawba County (link via MacRumors). The central server area is enclosed by 262,328 square feet of space supporting mechanical and electrical systems to support power and cooling. The building also has 57,432 square feet of space for offices and loading docks, and has an exterior equipment yard that covers 141,806 square feet. The building’s height ranges from 24 to 38 feet.”
Maybe some one already posted the google map link ( shortened )
http://goo.gl/IUE1H
Is that a power substation just for this building?
Might they even have multiple OC192 lines to service the location?
What ever Apple is planning to do or is doing, that is one enormous facility!
Coud it be possible that Apple will be bringing manufacturing back to the U.S. from China?
Hardly, since Apple is predicting China will soon be their biggest market (unless US wages fall below Chinese wages).
http://blogs.computerworld.com/18517/apple_crazy_china_will_become_biggest_iphone_market
Speaking during Apple’s Q2 financial call, Cook said: “Greater China saw iPhone sales being up over 3x, about 200 — almost 250%. And this catapulted revenue for the first half or first fiscal half in Greater China to just under $5 billion, which is up almost 4x year-over-year.”
Dear Robert,
I think you should talk to your shrink about upping your dose.
Sincerely,
Mike
You’re like the Glenn Beck of “tech columnists”.
Hope all the attention/page hits are worth it to you.
I think you forgot the fact that a lot of the computing power they will be utilizing will go towards analyzing what people are viewing, keeping their places of movies they are watching across multiple devices, and making suggestions of what else they would be interested in. Also, keep in mind that they probably have data backed up a few times, and they are probably doing point-in-time snapshots (ala time machine) of people’s data. It may not fill up right away, but it probably eventually will.
I was wondering — do you use vaseline, or just have the tinfoil hat directly on your head?
What the hell is he building In there?
He has subscriptions to those magazines.
He never waves when he goes by…
Maybe it’s a marijuana grow op…
Medicinal, of course 😉
Wile E. Coyote
Genius!
“Cringley’s estimate that the Apple facility could hold 7.2 million servers is based on the notion that Apple has 1 million square feet of space that can house racks of servers. The problem: The Apple data center is 500,000 square feet, not 1 million (a fact that has been widely reported). The equipment area – the only space that can actually hold racks – makes up just 184,000 square feet of that space, according to planning documents (See our Apple Data Center FAQ for more details).”
Quote from:
https://www.datacenterknowledge.com/archives/2011/06/29/cringley-on-apple-the-math-needs-some-work/
I hope you’re the one joking because otherwise I’ll have to think you’re just an idiot.
Touchy, touchy, aren’t we?
Well 200 million iOS users with each 1000 photos in their photostream, each at 3 MB is 600 PB (petabytes 10^17). That’ll fill up a couple of racks…
There is stuff in that data center, but I bet its half full or less. Apple is planning ahead. Its not a facility built to intimidate other, though it probably does. Its built to allow apple to expand as icloud and other services and the company expands. Apple has been renting space and running a mishmash of datacenters for years, much like the large number of different buildings they own lease and rent here in the silicon valley. This building and the two others they have hinted at looks like they have a large datacenter plan designed to grow with the company, much like they are building a headquarters to finally hold the company under one roof.
I bet apple is working on an overall housing plan for the entire company to handle growth in an elegant manor for the next 10 or 20 years.
Ah yes, the “folks at the convenience store down the road.” Did they also tell you about how 9/11 was a holographic projection designed to distract us from JFK’s real killers?
Bob, the very fact that your analysis doesn’t mention the words “iCloud” means that it’s hard to take it seriously.
Let’s look at it again.
You claim each of 700,000 movies, each at about 5GB each.
Well, iCloud gives each user 5GB of storage. Each of 125 MILLION (currently) iOS users. Let’s throw in around 25 million Mac users — maybe not a great stat because there is overlap, but give us order of magnitude.
That means we’re talking 200 times your base storage case — we’ve already used up your two 10x fudge factors, with an extra factor of 2 thrown in.
Of course we expect that (at first) most users will not be close to that 5GB of storage. But the point is, you are COMPLETELY looking at the wrong part of what is going on.
The second point is that what Apple is doing here has the potential to be very different to anything that has been done before. iCloud is NOT a file system in the sky, Apple’s version of DropBox. iCloud is, rather, an infrastructure (meaning both storage, but also code and APIs) for allowing apps running on multiple different devices (think iPhone, MacBook, and iMac) to maintain their state and storage in sync across all these devices. What state and sync? Well, Apple doesn’t know for sure — that’s why they’ve provided a rich set of APIs, allowing apps to use this remote storage in a variety of different ways.
The point is that other large data farms (think Google) are primarily read-only stores. They do some writing to maintain stats and stuff, but read is where it is at. This store is essentially read-write balanced — with the expectation that (unlike GMail say) we could be seeing hundreds or thousands of small WRITE updates from each user every day.
And it doesn’t end there. We have no idea, for example, if Apple have plans to offload COMPUTATION to these farms. The obvious candidate here is better voice recognition.
I’d recommend you should go all-in on this line of thinking. Apple has claimed that, not only do they have this massive data center, iCloud users around the world will utilize “local” data centers. Local was not defined, but presumably means something like, at the least, a Euro center, at least one Asian center, and maybe a US West coast center. Why not do some research into whether these are also phantom? Perhaps the Asian center was subcontracted out to SinoForest?
West Coast center was confirmed last month to be in Santa Clara. East Asia and Europe are no brainers for the next ones as you say. After that it might be interesting to see where the next ones (if any) go… Shanghai, India, Latin America?
I’m afraid the data center will be quite operational when iOS 5 arrives…
[…] Häpnadsväckande, om det stämmer: So I drove over, took some pictures, and talked to folks at the convenience store down the road. My conclusions from this unscientific research is that the giant Apple facility is mainly empty. It’s a huge building filled more or less with nothing and why Apple built it that way frankly escapes me. Maybe it’s just a shot across the bow of Google and its $650 million data center in South Carolina. […]
Unscientific indeed! The only valuable thing here is Cringley admitting that this is scientific. Cringley has lost it entirely!
Apple is building with the future in mind. Always. Strategic thinking and execution on goals you (and many of us) do not yet comprehend.
The cloud is your new hard drive. Like the floppy, HDD will go the way of the dinosaur. Minimum solid-state memory and a big honkin’ disk in the sky is what we’ll all have.
Apple knows it and they are building the solution to provide that to the masses, now, and 20 years from now. No one else in the business is doing that.
Ever think that Apple will be able to hold some very aggressive price points once competition finally steps up?
That’s no datacenter; it’s a space station!
– GammaRay Rob
The facility is 500k square feet (http://arstechnica.com/apple/news/2011/02/apples-nc-data-center-coming-online-this-spring.ars), not 1 million. It only contains about 184,000 square feet of data center space.
No one builds a data center to meet their immediate needs only, a typical facility will contain enough floorspace to support 5-10 years of future operations (at least). The facility will typically energize and pressurize space as needed, so yes a lot of it may well be empty and unpowered.
Your IBM olympic coverage analogy was just awful and totally irrelevant. You mean IBM used less space to broadcast a single event then Apple will use to store data for 100s of millions of people? Really? Wow. Why does Ford use such big factories when the guy down the street built a bike in his garage?
I am also not sure why you think the 3.29 petabyte number is apropos of anything. That is not all, or even most of what this facility is for. Apple just purchased 12 petabytes of raw storage (6-9 petabytes usable) and that was just the initial buy of one type of storage.
Cringely , Keep up the great work. Just kidding. Please stop reporting this kind of trash journalism. If you do stop, thanks in advance. -The World
Wow, this is about the dumbest piece of drivel I have ever read. The only joke is the writer of this crap. There is no need for Apple to build a huge building to scare the competition, they are about to issue out millions of free mobile me accounts. And they are probably planning for future expansion. That of which I am sure not too many people know about considering Apple’s secrecy.
I suppose you can now use this article as an excuse to write off the trip on your taxes, Good job!
Maybe their accountants said Supersize NOW we need the tax rightoffs!!!
Yes, I’m drunk..
I mean writeoffs burp.
If there’s too much space – What about manufacturing?
Looks like Dalle did some math… too.
**QQ** According to the Internet Movie Database there are about 700,000 movies in existence, excluding porn. Most movies will fit (in DVD form) in 4.7 gigabytes. Do the math and you get 3290 terabytes… **/QQ**
200M iOS 5 users (just for ball parks) and 5GB x 200,000,000 = 1,000,000,000GB of data … for their photos. There is music – which is sorta replicated (I would imagine via a db via iTunes), but it does take up space and of course there will be redundancies and backups. Say NC backups SV …. or Cupertino or something like that … Not saying they to a full nightly backup mind you, but probably a differential or something.
Yeah, its a pitty Apple has lost its brilliance. They could have been good – or even Great on both sides of the fence – for techies and non techies – since OS X is based on BSD/NeXT and their server OS was pretty good … maybe not nearly as scalable as Solaris, but damn, OS X wasn’t bad either. No more LOM, no more rack mounted servers from Apple, no more management of OS X clients via OD …. via rack mounted servers. No more external disk array management… hell even the internal MEGACard arrays were kinda cool with their CLI management. It’s ALL in the cloud, baby – for better or worse.
In terms of numbers of comments, this week’s column was a hit! Even the whiners showed up to play today. 🙂
I think they’re going to sell us all wearable computers, at a high price that we’re more than happy to pay, and they’ll crowdsource the ongoing digitization of the world in a way that will make google street view look like a poorly illuminated manuscript from the 12th century.
“I think they’re going to sell us all wearable computers…”
No, fashion faux pas, obviously.
” …at a high price…”
Wrong, the market is going cheap. “6 Android activations per second” and slated to double in the next few months.
“… we’re more than happy to pay…”
Nobody sane is ever happy to pay. See previous retort above.
“… and they’ll crowdsource …”
steal from the unsuspecting
“…the ongoing digitization of the world… ”
Somebody else’s idea and many others are already doing this.
“…in a way that will make google street view…”
Oh, the planned “Apple Alley View”? Who is going to use it? Thats almost as ridiculous as Ariana Huffington being in charge of Mapquest.
” look like a poorly illuminated manuscript from the 12th century”
Now you’re just being mean and disgusting. Its hard to link any thing Apple to religion.
Bob,
thanks for your great post, it’s interesting as always. However there is one fact, that bothers me, your estimations of how much space it will take for hosting all the movies is correct. But as my experience shows (and I have more than 6 years experience in online video streaming) it takes more than one server to serve a movie to let’s say a 1000 users. If you factor in the average speed of the stream you can hit a limit of the LAN very fast (if you have 1Gbit lan, or bonded 2 Gbit LANs). Based on what we have seen you can stream at most 8-9Gbps from 1U server with SSD disks and 128Gb of RAM and not more than 3x128Gb of different content (because of IO overhead and OS limitations). So when you factor that in your calculations should be revised.
Good investigation. But maybe time to get yourself an unmanned drone? 🙂
Good idea, Cringely drone cloud! What drones do you prefer, my man, Predators, Raptors, or Reapers?
Um, on a more serious note, my condolences to you and your wife on her father’s death. Any parent’s death is a hard blow to take. My best wishes to you and your family.
Parrot AR.Drone anyone? What’s its range?
so in other words you’re saying you couldn’t get there, couldn’t see anything and then you based this article on that fact and wrapped it in your ‘estimates’ based on nothing. is that correct or am i wrong?
You are wrong. Robert made a point of noting that this facility essentially had little to no activity from his direct observations and from speaking with the locals. Also, remember that you’re on cringely.com, an OP-ED by Robert X Cringely. He’s free to make any conjectures he makes.
Another route to analyse all this is economics.
If the facility *was* filled with typical servers, and each server had typical network utilisation, how much would it all cost Apple? Is that number feasible given Apple’s revenues?
Both Google and Apple are public companies and must publish accounts. Are Apple’s number for running servers ten times those of Googles? I think it is safe to assume that Apple are no better at running servers efficiently than Google are. So if Apple’s expenditure on servers is about the same as Google’s, that would indicate the facility is indeed 90% empty.
It’s not 90% empty, it’s used for growing pot: “we need all the electricity to (ahem) run servers, you know”
Nah, you mean iPot right?
I’ve got the same impression without even visit this data center. I’m iOS developer and I see almost every day sail reports in AppStore and almost every day I see message like this Please note: iTunes is aware that report generation for AMR is delayed for 21st JUN, 2011. We apologize for any inconvenience, reports will be published as soon as generation is complete. And they still host service for developers, like iTunes Connect on the Akamai Technologies servers. So, I’m pretty sure that this data center is empty.
> And they still host service for developers, like iTunes Connect on the Akamai Technologies servers. So, I’m pretty sure that this data center is empty.
Akamai is a CDN. They cache data on servers all around the world which are very close to end users. A data center is something very different, it provides data to the CDN and end users.
Just point that out, not saying it isn’t a hoax. It possibly is and I also was thinking of the drone reconnaissance idea.
Sorry sale reports, not sail. 🙂 I’m also yachtsman 🙂
Do they keep the SAT test papers in there? Could be a perfect score. Or only just a dream.
You could have seen more on the Google earth view. 12 vehicles, some look like white maint. type. Those two huge tanks and all the AC along the back. A pond in front. Could be for looking pretty in the future or cooling water. Ask your expert friends about exposure time to confirm that none of the AC fans are moving. You could also ask then about the time the photos were taken to see if it was open, yet. The trees are bare but even in winter data centers need some cooling.
Not necessarily. My company, NetApp, built a data center at its campus in the Research Triangle Park outside of Raleigh. It uses outside air cooling. On the tour I was taken on, they said they don’t have to turn on the AC until the temperature rises above 85 degrees. The only part with AC and heating the rest of the time is the fancy conference room with the glass wall called the executive briefing center. Presumably Apple built a state-of-the-art empty warehouse, so the state of the AC fans doesn’t necessarily mean anything.
What Andrew Robinson is suggesting is very interesting: they’ve built a state of the art data center which mainly doesn’t need AC. It’s empty on purpose. Take that, Google.
Isn’t Hotmail the largest email service as opposed to Gmail?
Gmail passed them this last year, I believe.
No, they didn’t. They might have just passed them in traffic, but that’s mostly a search figure (a lot of people use Outlook or other desktop clients to check email). GMail has ~200MM users; Hotmail has ~500MM.
My company (which shall remain nameless) has some quantity of data centers scattered around. The newest one (which they picked up for a steal from a dot-com failure), is fully “Twinned”. That means there’s two of everything — One of each item is fully capable of providing for the whole site for a significant period of time, and there’s two of them. That means that they can take one system offline for repairs and they’re still in business.
There’s two power drops coming into the complex — from different power grids. There’s two battery banks. There’s two sets of diesel generators. There’s two sets of diesel fuel tanks. There’s two sets of piping from the tanks to the generators. There’s two sets of air conditioners. There’s two sets of water chillers. There’s two sets of chilled water holding tanks. There’s two sets of pipes providing chilled water everyplace. (If the data center is in a cold enough place, there’s two sets of heat exchangers which can chill things using outside air.) There’s two sets of fire extinguishing systems. All that takes up space.
I believe I estimated that one third was data center, and two thirds was support services. So, of that million square feet you mentioned, probably only 300K sq ft is data center, 600K is the support systems, and 100K is office space for the people who have to keep all of that running.
Oh, yeah, I forgot. There’s two connections to the Internet, from different companies, coming in thru completely different sets of cables along different paths. All the stuff you need to make sure that, barring a direct nuclear strike on our facility, we can not go down due to something mundane.
It may be something totally different.
I think it’s going to be a repair center. If you’ve ever tried to get a Mac, iPhone, or iPad repaired it takes WEEKS just to get an appointment at an Apple store then more weeks to get it repaired. It really sucks.
They need a centralized facility to streamline repairs – I think that’s what it may become – along with a data center.
That’s my theory & I’m stickin’ to it!
🙂
can’t be a repair center. that stuff would be farmed out to whatever island is above water this week, like everybody else does, for competitive reasons.
But the turnaround time would be less!
Um, it doesn’t take weeks to repair an Apple product (or get a GB appt for that matter). Apple’s depot locations (in Houston and Nashville), for portables, are on average 2-4 business days. Most reputablely staffed ARS locations crank out desktop stuff in 2 days or less. Same-day repair percentage is very high for Apple, regardless of whether in-store or mailed out. Transit and waiting for parts can cause delays however.
As I recall, Google used to go for the oldest commercially available hardware they could find – P3’s even after the rest of the industry had moved on – because the old boxes used far less power than new boxes.
Just serving video and audio doesn’t take a lot of brute force – just storage and pipes, for the most part.
So maybe that’s it.
That, or perhaps as Gruber says: you’re FULL-nuts now, instead of half-nuts.
My condolences to you, your wife, and your family.
Bob – A few weeks ago you wrote:
“Notice I didn’t mention Apple. In terms of being the baddest MoFo in the market Apple has no peer, but Apple is following its own very different course. Apple isn’t the next Microsoft, you see. Apple is not the next anything because the role it aspires to transcends anything imaginable by Microsoft, ever.”
I thought that comment was brilliant and was anxious for you to elaborate. Are you now saying they aspire to be an elaborate, expensive hoax of a shell company that intimidates competitors based on empty buildings and a pile of cash?
It’s still not large enough for Jobs’ ego.
Ah, finally, a reasonable explanation! “My data center is bigger than your data center!!!”
Could have done a search instead for a aerial shots of the data center….
Another link, it’s still speculation, but better than sitting in a car, on a road, near a gated site…
https://www.fastcompany.com/1752558/voice-recognition-for-iphones-floats-apples-data-center-into-starring-cloud-tech-role
Apple may need this facility as a negotiating tool and/or a fall back facility in case there are earthquakes in somewhere like Japan or China. “Without the 10% discount on future builds, we have a factory ready to go in the USA.” Just an idea.
This all assumes Bob X. has done his data center math correctly. Nice thinking Bob. It got me to thinking.
> Following the recent death of my father-in-law
My deepest sympathies to the entire Cringely family, and especially to Mrs. Cringely. I just buried my own father this week. It is a very hard time.
There is a lot that you didn’t account for …. just the free 5GB storage for iCould would require 500 Exabytes (yes, that’s 500,000 Petabytes); this is based on 100M active iPhone/iPad3G users. Add to that, the free Photostream (could be gigantic), all the free music serving (entire iTunes Library), plus “paid” storage.
You’ll need MASSIVE computational power to serve and backup this content. You’re not thinking big enough.
Also, Apple is a such a secretive company, that they would never let anyone know what they’ve got planned for this facility. There are probably hundreds of different services they’ve got coming down the pipe. It’ll probably start modest with iCould, and eventually expand out.
Finally, a rational and well reasoned explanation for scope of the datacenter. All through reading this article, I was wondering if/when Mr. Cringely was going to realize that iCloud (not ‘iCould’, BTW) will be much more than Apple’s version of Netflix, and correct his half-baked analysis.
I have to go with Gruber on this one. Somebody may need psychological evaluation – or just to cut back on the caffeine intake.
All joking and jostling aside, I also wish the entire Cringely family my deepest condolences and wish them well.
Sorry to hear about the family problems, Bob. A bad time. 🙁
Do you remember the old joke that circulated.. must be early 70’s? Not quite sure now. “Did you hear about the Japanese electronics company? It was so successful it has had to move into smaller premises!”
Based on your estimates that building really shouldn’t be a data-centre, should it? Maybe something else is planned… Electric cars? The next Apple “big thing”? What are taxes like on empty land/building use in the US?
If it’s just a joke then what a sense of humour! 😉
“It was so successful it has had to move into smaller premises!” If you think about it that applies to all human endeavors aimed at improving efficiency.
Storage: You mentioned 329 TB of data.
I’m pretty sure Apple has put at least 10PB of data capacity into that facility. I can’t say where I received the information, but I have been told that the hardware is from Isilon.
I am going to say that Apple has a much much larger vision than just storing a few movies. I have some iDevices. Besides playing games and reading books, I do social networking and search, as well as video and pictures from my iPhone. I’m sure Apple would prefer I did those activities from them and not from Facebook and Google and Dropbox. As you say, they have the cash. But Apple never does anything for free, so we’ll see if they can ride that 30% margin they have over everybody else to the finish.
As for each Rack, I set aside at least 16 Sq Ft per rack. That includes the rack (approximately 8 Sq Ft) and the space in front and back of the racks for the aisle. And I go for the extra tall racks, 52U at least. Find a Ladder!
Really now, even those of us who don’t do tech punditry for a living know that iCloud is more than just streaming video.
But to be fair, the capacity estimates, as bad as they might be, tells us that on cloud connectivity, Apple is thinking big, like really big, as in beyond our wildest dreams big.
The problem with that is, even if they are thinking BIG, Moore’s law still applies. Google can INDEX the WHOLE internet with 13% of that total volume. So even if 100,000 SQ/FT is only for actual “servers” and storage they still would only be using 1/3 of what they have so what are you doing with the other 200,000 SQ/FT of the space? Even if your cashing all of iTunes/ user devices data lets assume 100,000 SQ/FT, 100,000 SQ/FT if they want to cash the WHOLE internet. That leaves 100,000 SQ/FT for what? Jobs doesn’t do anything without a reason…
I live just down the road from this facility. There is plenty of “traffic” that goes in and out at normal shift hours. If you pick one hour in the middle of the day, no one is going to go in or out. Companies don’t appreciate when dozens of their employees just decide to leave for no reason, go hang out, and then come back. Also, it is not a “corporate” facility. No big management meetings really take place there, its just a farking data center. I think the author of this article is a hipster who can’t stand apple. I’m not a huge apple fan, but you’ve got to be unbiased when reporting news, which by the way, this isn’t.
It’s run enirely on the power of the smug coming from their customers.
Possible other explanations:
1. No, it’s run entirely on the power of the smug coming from Prius owners;
2. No, it’s run entirely on the power of the smug coming from a recent Nobel Peace Prize winner who’s running for government office;
3. No, it’s run entirely on the power of the smug coming from android users;
4. No, it’s run entirely on the power of the smug coming from Crawford (see how I did the equal opportunity thing with #2?);
5.No, it’s run entirely on the power of the smug coming from people who reply to other’s comments in tech blogs (hey, wait a minute).
It’s to hold Apple’s excess cash.
I like how people keep throwing servers around like they are the most efficient way to store data. If this truly is for iCloud, it’s most likely to be a ton of storage appliances (generally 6U high) and storage disks to go with them (3U to 4U high with several per storage appliance). You can’t look at a storage facility as a whole bunch of 1U servers. It doesn’t work that way. As a storage systems admin at NetApp, I can tell you that the space requirements for large volume storage get very big very fast.
Two things – your rack calculation is off, and it doesn’t matter anyway.
Racks:
Our rule of thumb for data center floor space is 15 square feet per rack. Yes, most cabinets are about 2x3ft or a bit bigger, but assuming an extra 4sqft means you have 2ft in between rows, which isn’t a lot of space to get gear through and into the racks.
Also – I don’t know where you got your 1Msqft number, but for most data centers, about 1/3 to 1/2 of the actual pad is data center space. 1/3 gets you a standard data center with air handlers out on the floor, although the usable space could be less if the building isn’t designed efficiently. For a facility that puts cooling units outside (see Switch’s Supernap in Vegas for an example, or if you have minimal cooling due to outside air use) you can get higher than that.
So your numerator is probably less than 1M, and your denominator is bigger than 10.
So what, no one cares about racks:
Anyway, screw all that. What you ought to do is go find the power contracts, which are probably public record, at least for most counties. Square footage doesn’t mean crap in any of my data centers – it’s all about how many Watts I can throw to my racks and then back up with UPS and generator. So figure out how much power they have committed to them and it’s a lot easier to figure out how many servers they are planning. Remember that for every watt of energy you put in, you have to extract it – so unless they are doing something crazy efficient, maybe half of the committed power may be going to cooling to extract the other half that goes to the equipment.
try this link from 2010.
http://tech.fortune.cnn.com/2010/10/27/apples-invisible-server-farm/
Man, look at the place on Google Earth. Apple ruined some nice countryside. And probably pissed off that neighbor in the secluded house to the West — right across that new array of transformers and power equipment.
But… my problem is that it’s completely nondescript. Which ruins the argument of the article. If it was meant as a show-off building, it should say “Apple” on the roof in huge letters.
Also, the lack of traffic may mean nothing. There is almost no parking (compare to the immense lot of the Target Distribution Center up North a bit – another white box building). And that makes sense because servers don’t need people around.
Ultimately someone who lives nearby needs to explore.
[…] Have you heard the one about Apple’s data center? […]
It’s most likely an automated “lights-out” facility, remotely managed with only a dozen or so staff nearby for manual intervention.
The Carolinas have a long tradition of shift-work. So I suspect that the during hours that Bob was there, it was in the middle of a shift. At shift-change, however, don’t get in their way, they’re heading home – fast!
Since this datacenter is single-vendor, you don’t have a lot of people showing up in the middle of the day to go do things in their cages. Sure, there were lots of people on-site when it was being populated with hardware, but now that it’s presumably complete, only maintenance staff are needed for “remote hands” work.
Bob – Have you considered Austin? Yes, it’s hot here, but not as humid as Charleston. And you don’t have California’s teetering economy & taxes.
I’m curious how much traffic Cringely expected to see coming in and out of the data center. Since nothing is fully operational at the moment, there is no reason to have a full staff on hand now, and that staff figures to be in shifts for a 24-hour facility, anyway.
The Google Earth photo reveals no more than 60 parking spots on the entire property, and a percentage of those figure to be meant only for vistors. This center, when it is at full throttle, was supposed to employ 300 people total (though if the company goes forward with plans to duplicate the storage facility in the adjacent lot, that could bring a few more jobs to Maiden). Right now, though, it is doubtful there is much more than a skeleton crew on site. When iCloud comes online, there figures to be more activity there.
Unless there are parking spots inside, or even underground?
On both sides of the front part (the white rectangle just on top of the outside parking lots), there could be some kind of entry roads. Maybe there are even ramps to go underground?
OMG! it’s Apple’s skunk works. … It’s Area 51a.
This is where Apple furtively dream up and covertly develop all those grotesque and immoral fruity products.
Those products that seem to hold an ever increasing number of people around the world iTransfixed.
It’s a conspiracy I tell ya’.
Chris’ comment of a couple of days ago was fundamentally right..the size of the building and the real estate costs are not hugely important. First, the thing you need to look at is how much power is available, how it is priced, and how reliable is it. Second, real estate costs are a fraction of the infrastructure costs, so the best strategy is to buy a lot of room if you have the capital and the power in place. Third, the build out inside – the real cost – will be modular, scaling gradually on demand. The size of the shell isn’t that important, its just cheap headroom.
[…] according to Robert X. Cringely’s bog, the place has no equipment in it and is mostly […]
[…] a bit, first let’s take a look at the straight facts from Data Center Knowledge… Next, let’s have a look at the conspiracy theory from tech crackpot Cringely… and then back to the DCK for a check on Cringely’s […]
[…] according to Robert X. Cringely’s bog, a place has no apparatus in it and is mostly […]
I do not know of a Google Data Center in South Carolina that may irritate Apple but there is one in NORTH Carolina, in Lenoir, about 30 miles northwest of the Apple site. Maybe an accident of geography and resources, maybe not.
It’s for real, Teradata database machines, HP servers and NetApp NAS. Looks like they got a deal from Duke Power and got the city to build a fire station across the street.
https://www.theregister.co.uk/2011/06/09/apple_maiden_data_center/
https://www.wcnc.com/home/Why-Duke-Energy-Stands-To-Make-Millions-From-the-iCloud-123473304.html
First, Bob we will miss you here in NC.
I’ve been to Maiden and Bob’s account is spot on. Even a year ago when it was still under construction and tough to see anything.
Thought to Ponder:
15 years ago how were you cataloging your life?….. chances are it was on film. Today we point and shoot and upload them stream, and we do this in HD. Forget commercial films and music for a moment and contemplate how much each person will need in total and you’ll start to understand how the case for a second building even would make sense.
-MacHunterz
Did we include space for the indoor go-kart tracks and bowling alleys? Jus’ speculating…
Perhaps it’s Steve’s PYRAMID !!!
There’s two power drops coming into the complex — from different power grids. There’s two battery banks. There’s two sets of diesel generators. There’s two sets of diesel fuel tanks. There’s two sets of piping from the tanks to the generators. There’s two sets of air conditioners.
[…] took some pictures to get the lay of the land. Here’s his assessment of what he saw, on site. “Have you heard the one about Apple’s data center?” Bob is a pretty astute observer, so his findings are a bit […]
[…] taken while the place was obviously still under construction, and a recent photo and write up from Robert Cringely on his blog, but I didn’t actually finds those until after my wasted trip. I can tell you that the place […]
[…] Perhaps he is looking for something to fill the servers in Apple’s new server farm that Cringely recently reported appears to be […]
https://www.onepearls.com/bracelet-jewelry-shell-bracelet/c13_61/index.html
You’ve answered your own question: Apple is probably planning a soup-to-nuts alternative to Google. It’s the only thing that could require so much space.
Apple is one of the only companies with both the motive and the scale to pull it off. Android is a threat. If Apple search becomes the default on every Apple device, they potentially cut 20% from Google ad revenue. If you look at GOOG’s cost structure, this is enough to change their economics.
Reminds me of Oracle’s 1980s-era “cut off the oxygen” strategy.
Developed in collaboration between legendary music producer and artist Dr. Dre, engineers from Monster Cable and renowned industrial designer Robert Brunner. The Beats by Dr. Dre Studio headphones allow you to experience music the way the artist wants you to. These high definition headphones are precision-engineered to reveal the full sound of today’s digital music including the most sonically demanding rock.
It uses outside air cooling. On the tour I was taken on, they said they don’t have to turn on the AC until the temperature rises above 85 degrees.
[…] planning for the future. They already have an empty $1 billion data center in Maiden, NC. … Nobody knows what they are going to do with it, perhaps not even […]
what is kindle?…
I, Cringely » Blog Archive » Have you heard the one about Apple’s data center? – Cringely on technology…
[…] empty building to intimidate its competitors. This wild theory comes via Robert Cringely, who did some math back in June and figured that Apple’s data center was far huger than it needs to be even if Apple wanted […]
[…] empty building to intimidate its competitors. This wild theory comes via Robert Cringely, who did some math back in June and figured that Apple’s data center was far huger than it needs to be even if Apple wanted […]
Just a correction point, Yahoo has the biggest email service 275+ million users, not Google.
[…] we have the second observation that fits in with the first rumour. Back in June Robert Cringely wrote blog that seemed amusing but slightly crazy. He did some sums and worked out that […]
[…] options open.Then we have the second observation that fits in with the first rumour. Back in June Robert Cringely wrote blog that seemed amusing but slightly crazy. He did some sums and worked out that […]
[…] andatelo a dire a Robert Cringely, secondo cui l’immensa costruzione voluta da Cupertino è irrimediabilmente sovradimensionata anche se servisse per offrire in streaming on-demand ogni canzone e ogni film mai creati al […]
[…] een theorie komt van techjournalist Robert Cringely die beweerde eerder al dat het megalomane datacenter, ter waarde van 1 miljard dollar, van Apple […]
[…] empty building to intimidate its competitors. This wild theory comes via Robert Cringely, who did some math back in June and figured that Apple’s data center was far huger than it needs to be even if Apple wanted […]
Very good blog! I’ll come back here again!
[…] a huge data center investment, however – and Apple has already started down this path. Some reporters have commented how Apple’s Maiden, NC Data Center is grossly overbuilt, given what the iCloud […]
BTW the storage-system will be ISILON (now EMC).
FYI: http://news.cnet.com/8301-13579_3-20051364-37.html
in short: 12 petabytes
Now isilon isn’t exactly a hi-density storage arch…, so this will take
quite a few racks – but yea – maybe 30 of 40U racks…
hmmm…. funny
[…] Amazon crushes it with its ready-to-go access to all those resources in its cloud. Apple will get there, too, eventually. See […]
I’m sorry, Cringely, I have to disagree on your assertions. Obviously the data center was built to house Steve Jobs’ new brain – I doubt THAT technology can fit into an iPad already.
That’s some pretty nice reporting on the Apple Datacenters out there. I wish I had come across the photos when the news was fresh. I’m sorry about the death in the family.
amazon cyber monday…
[…]I, Cringely » Blog Archive » Have you heard the one about Apple’s data center? – Cringely on technology[…]…
Hi my friend! I wish to say that this post is awesome, great written and include almost all important infos. I’d like to look more posts like this .
Security Chronicles, Security Guard, Funny CCTV, Funny Security Footage, Security Stories…
[…]I, Cringely » Blog Archive » Have you heard the one about Apple’s data center? – Cringely on technology[…]…
Hi, i believe that i noticed you visited my blog so i got here to go back the choose?.I’m trying to find issues to improve my site!I assume its good enough to make use of some of your concepts!!
Excellent post over again! Thumbs up!
I just like to say it how I see it. Take care.
[…] What’s interesting about this is that Apple’s existing iCloud data center in North Carolina potentially offers server capacity to rival Google. “Apple could get away with a tenth of the space to run its iCloud at current capacity…Meanwhile Apple is running chunks of its iCloud on Microsoft Azure and Amazon S3,” notes TechEye, citing i Cringely. […]
[…] What’s interesting about this is that Apple’s existing iCloud data center in North Carolina potentially offers server capacity to rival Google. “Apple could get away with a tenth of the space to run its iCloud at current capacity…Meanwhile Apple is running chunks of its iCloud on Microsoft Azure and Amazon S3,” notes TechEye, citing i Cringely. […]
bao da iphone 4…
[…]I, Cringely » Blog Archive Have you heard the one about Apple's data center? – I, Cringely – Cringely on technology[…]…
Monster By Dre Spiderman…
[…]I, Cringely » Blog Archive Have you heard the one about Apple's data center? – Cringely on technology[…]…
search engine…
[…]I, Cringely » Blog Archive Have you heard the one about Apple's data center? – Cringely on technology[…]…
interior design…
[…]I, Cringely » Blog Archive Have you heard the one about Apple's data center? – I, Cringely – Cringely on technology[…]…
Another stone in the mighty citadel of Apple!
A number of excellent content pieces here on the subject of just how to take good care of your white gold locket
[…] may recall I wrote a column a couple years ago about Apple’s huge North Carolina data center. This was just before we moved back to California. In that column I questioned whether anything was […]
ways to make money from home
[…]the time to read or stop by the material or internet sites we have linked to beneath the[…]
Ariana Grande Naked Photo
[…]always a significant fan of linking to bloggers that I like but dont get a good deal of link enjoy from[…]
penambah berat badan tiens
[…]the time to read or stop by the content material or web-sites we have linked to below the[…]
Surf City NC DJ Service
[…]just beneath, are several totally not associated websites to ours, having said that, they’re surely worth going over[…]
https://www.adsbeta.net/uHFGY
[…]usually posts some quite fascinating stuff like this. If youre new to this site[…]
garage door
[…]below you will obtain the link to some sites that we assume it is best to visit[…]