or: How I Haven't Learned to Stop Worrying and Love the Cloud.

there's a persistent background of fear in the Apple community. even when rationality ought to prevail (as it most certainly did in John Gruber's written and spoken coverage of the Mountain Lion announcement), paranoia rushes in to fill the void. "it's only a matter of time", the devil perched on your shoulder says, "before they take away the filesystem. for good. everywhere."

an aside for all those about to go Van Hœt on this: what we're talking about is the filesystem metaphor, i.e. that the stuff on your computer is stored in files which get organized into folders. this is separate from the filesystem qua filesystem, the method by which a computer actually stores, organizes, and finds data on disk. Hypercritical e56 is a nicely comprehensive review of these, and from 1:42:00 to the end deals with the actuality vs. metaphor distinction. to disambiguate, John Siracusa uses the term "file hierarchy", which i think is certainly apt, and points out that one of the major reasons that its demise might be imminent and even welcome is the fact that most people simply cannot grok it. this is certainly true. ten years on, my parents still haven't quite grasped where /Applications is in OS X. they had no trouble with the file hierarchy in Classic Mac OS — just bung stuff wherever you want it, as long as you don't touch the System Folder. so the question is, if the file hierarchy ceases to exist, will they lose anything? i think the answer is yes, but perhaps less than most people think, and for a different reason than most people think: namely, that we've already lost a good bit of the meaningful hierarchy.

the case of the missing calendars

last fall, like many people, i updated to iOS 5. also, i wasn't an early adopter of Lion — i waited until 10.7.2 came out and then did one epic afternoon of software updates. i mostly did this to try to ensure a smooth transition to iCloud. despite having to actually set up iCloud from the bleachers at a hockey game because i didn't believe that the full wipe and restore of my phone would take as long as iTunes warned me it would, everything seemed to go perfectly smoothly. until a couple months later.

one day i was idly playing around with the ⌘4 / heat map / year view in iCal and noticed that 2009 and 2008 looked a little … sparse. (my iCal universe begins on April 6, 2008, due to massive a hard drive catastrophe that happened then. back up, kids!) after digging a little deeper, i noticed that some calendars, despite professing that they were merrily synced to the cloud, had zero events in them. oof. for whatever reason, iCloud had been benevolent and only disappeared the contents of calendars that were no longer in active use; otherwise it would have been a code red disaster. as it was, it was an inconvenience that left a hole in my personal digital history, which i now try to preserve a little better than i did in March 2008. i filed it away as a problem to be dealt with later. perhaps i did that because i knew it would be no simple task to get them back.

backups are only as good as the files they contain

what i was up against was another concern of power users everywhere: (1) the cloud is canon, (2) the cloud is now, (3) the cloud is only now. in this case, iCloud was very sure that those calendars weren't supposed to have any events. that's how things are supposed to be now, because those are the instructions the cloud received, be it in error or good faith. and iCloud has no idea if that's how things used to be, or whether those calendars ever used to have any events in them. the fact that i had a human, fleshy, grey matter memory that i definitely put things there and used to be able to look at them was really of no concern to the cloud. as i learned in the 2008 hard drive crash, wistful memories don't bring data back from the dead, they just make you feel like crap.

but all was not lost. those now-cloudy calendars were once local calendars, back when the canonical version lived inside my apartment, instead of in North Carolina. and i have Time Machine backups, so they must be preserved. it's just a matter of … finding them.

sidebar: i have considered myself a power user for a long, long time — my entire adult life, certainly. restoring a few files from a backup should be trivial, not throw me into a rage. but when there are as many layers of pain as there were between me and those files, there was no other possible reaction. bear in mind that this task would be impossible for an average user, and as i'll point out along the way, near impossible for a third party trying to help an average user. for a power user, it was merely maddeningly difficult.

ok, let's do this! first, where do those calendars live? must be somewhere in ~/Library …which is of course now hidden in Lion. no problem, ⌘⇧G will get me there in no time. from there it'll just be a quick hop into the Time Machine, rescue the appropriate calendars, and— oh, hell. ~/Library/Calendars looks like this:

we have just exited the universe of the filesystem metaphor. there is next to nothing here that is human-readable. perhaps i was supremely naïve to believe that my Typology calendar would be named something sensible like Typology.ics, or at least be a folder called Typology.calendar. instead i was staring down 32-character hex hashes. (note the Finder's hilariously awful attempt to sort these by name. since it tries to use logical base-10 numerical sorting, clearly a file that starts with 127 should come before one that starts 0259!)

i was not deterred. i went digging and found that each folder had an Events subfolder, which contained each event in its own self-contained .ics file. an empty Events folder indicated that i'd found one of my raptured calendars. there was also an Info.plist file. i started cracking these open, figuring that they had to include important metadata like, you know, the name of the bloody calendar. rinse and repeat for my 33 calendars. some time and scratch paper later, i'd compiled a list of dead calendars matched with the first four digits of their hash. now, certainly, i could step into the Time Machine.

since i access my Time Capsule over wifi, this is excruciating in and of itself. two minutes "Connecting to backup volume …". another three minutes remembering that any previous states of this folder exist, during which i can't do anything else with my Mac. (a necessary design choice when Time Machine was first introduced. but please, Apple, make this thing behave like a proper full-screen app in Lion.) all the while i'm not being soothed by my slow drift through some sort of nebula or star cluster, and the all-important navigation bar on the right side of the screen has rendered improperly on my external display. this is well past the point of giving up for most people.

stepping backward through backups proves incredibly slow and massively unhelpful — my calendars are modified between nearly every backup. so i get to play guess and check with when it was that i made the switch to iCloud. i finally find it! here we go, time to bring back those calendars when i realize, son of a bitch all the local calendar hashes are different. 33 new sequences of hex digits, totally unrelated to my notes and the previous work i've done. i am no longer surprised that iCloud lost track of some of my data.

this is also the stage where the endeavor becomes impossible even for a user being assisted by someone else, say a Mac Genius. first, the user would have to take their Mac and Time Capsule to the Apple store. then the Genius would have to get to this point. then, together, they would have to go through every calendar folder through the Time Machine interface, which is about as fast as working with floppy disks, handcuffed by only being able to use quick look to ascertain what's what. i found out that quick look-ing the .ics files was faster than the .plist files, so i did that. i wrote down another set of 4-digit hex prefixes on my scrap sheet, selected those folders and finally, at long last, hit Restore. after clearing the iCal cache and forcing it to rebuild its database, my events were back. not in the cloud, but back, on my hard drive…somewhere.

i don't want to have to be lucky

there's no question, i got lucky in this scenario. the snafu happened in the transition from local to cloud copies of my calendars. i had backups. i had knowledge and a hell of a lot of persistence. but what about the person who buys their first Mac today and always lives in the cloud? when their data goes missing, who's going to be able to save it? it'd be like waking up one morning to find that your car was missing from your garage. you call the police to report it, but once you give them your license plate number and VIN, they tell you that they have no record of the registration and that in fact, the vehicle had never even been produced by the manufacturer. this is nightmare, Kafka-type stuff.

everybody, power-user or not, should want to be able to lay a hand on their data, in both the literal and metaphorical sense: "look here, i hold in my hand a drive that contains my data" and "look here, i'm pointing with my cursor at the representation of that data". if all our data lives inside apps, all our apps have to be able to talk directly to our incremental backups. a couple Apple apps do this — Mail.app, for one — but the vast majority don't.

losing a file hierarchy might be a pain for some power users. but plenty of those power users use a flattened system for the one place they have more data go in and out on a daily basis than any other: their gmail archive. gmail tags are a flat storage system (ironically, to be compatible with IMAP, which uses folder hierarchies, google had to hack a flattening mechanism on top of the hierarchy. actually, the same is true for iCloud — anything saved on an HFS+ drive necessarily lives in a directory, even if it's invisible, unaccessible, and miserably named.) i am all for organizing my stuff in the most efficient way. even though i'm a file hierarchy junkie, i usually cut through most of the layers by using LaunchBar. my files live in a semi-flat representation in my mind already.

fortunately, i don't have to conclude this by predicting only doom and gloom. iCloud is gravely unsatisfactory in terms of keeping my data safe, because it's dumb about backups and previous states. thankfully, we have an alternative: Dropbox. no matter what happens in Cupertino, it's a relatively safe bet that Dropbox is not going anywhere in the foreseeable future. and they provide two very, very important features. first, file hierarchy is their bread and butter. even on a device where the local file hierarchy is totally hidden, like in iOS, any app with Dropbox access lets you browse cloud-based files and folders. but most of all, they preserve previous states of everything, for a full 30 days even on free accounts. if something gets disappeared — through a bug, through human error, whatever — they will not deny that it ever existed.

so is Apple protecting our data? my answer on a file level, and John Siracusa's answer on a filesystem level, is a resounding no. this is sad, but it may change. at least, for now, there are many clouds in the sky.