The dark side of Optimize Mac Storage: What you need to know if you rely on it

During the course of the past few days, it’s become clear to me that there is a serious architectural problem with how Apple manages files on the Mac with iCloud, and that design flaw can lead to extensive data loss.

If you have more data in your iCloud Drive storage than you have space on your Mac’s internal SSD, you stand a good chance of losing files, even if you back up with Time Machine.

Also: The best Macs right now

Why is that? What is actually happening to your data?

Strap in. We’re going for a ride.  

A gentle introduction to iCloud drive

In 2014, Apple introduced iCloud Drive, part of its iCloud (now iCloud+) service offerings. Like Dropbox, OneDrive, and Google Drive, iCloud helps keep folders on multiple machines in sync, and provides cloud storage.

Apple’s iCloud is specifically integrated into the Documents and Desktop folders (as well as Photos). When you enable iCloud Drive on your Mac, iCloud moves those folders out of the Users/[yourname]/Documents and Users/[yourname]/Desktop directories and into its own space managed under the directory Users/[yourname]/Library/Mobile Documents/com~apple~CloudDocs.

Also: iCloud backups get end-to-end encryption

In general use, you won’t notice the move, because MacOS sets up symbolic links between where the old Documents folder lived and the new folder that is now tucked under the hidden Library directory.

Two years after Apple introduced iCloud Drive, in 2016, Apple debuted the Optimize Mac Storage feature in MacOS Sierra. Optimize Mac Storage was designed to solve the problem of users with low-storage Macs who had accumulated a lot of files, or who were running out of space.

The idea is pretty simple. When Optimize Mac Storage is enabled, MacOS keeps the most recently accessed files on your Mac’s local drive, but deletes older files locally and keeps copies of them in the cloud instead. Since the older files are stored in the cloud, the theory is you can always get them back. 

Also: How to buy more iPhone storage

To make accessing all your files seem seamless, Apple replaces the full-sized old files with small stub files that take just a couple of hundred bytes instead of the kilobytes, megabytes, or even gigabytes that most files consume.

The trick is that as soon as you access one of those stubs, MacOS goes out to iCloud, downloads the file and presents it to you as if it had always been on your drive. If space is tight, MacOS also deletes another old file from your local drive, keeps a copy of it in the cloud, and replaces it with a stub.

When everything works, this swapping out of older files for recent files is a very useful way to make a lower-end machine with less storage seem to have a lot more available storage.

One last point before we move on: This swapping capability is really only available to people who have purchased the 200GB or 2TB iCloud+ plans. Macs with only 128GB of storage can make use of the $2.99-per-month 200GB plan, while Macs with 256GB or more can benefit only from the $9.99-per-month 2TB plan.

Also: Apple Mac Mini (2023) vs Mac Studio: Is M2 really better than M1?

The baseline M2 Mac Mini ships for $599 with 8GB RAM and 256GB storage. This is actually a heck of a deal, considering the performance of the M2. Apple has long shipped Macs equipped with far less storage than many of us would accept in our phones, but it keeps the barrier of entry to owning a Mac down. Today, no Macs are sold with less than 256GB, but Optimize Mac Storage is still usable for older 128GB Macs — especially if you own more than one machine.

What we’re told about backing up Macs

I’ve written before about the importance of a 3-2-1 backup strategy. On the surface, it appears possible to do this with native Mac tools. The idea of a 3-2-1 backup strategy is to have at least three copies of every file, two of which are on different physical devices, and one of which is located off-site.

With iCloud Drive and one Mac, you ostensibly get two copies: one is on your local machine and one is in iCloud. If you have a second Mac, you get a third copy, because iCloud syncs down to that Mac.

But there’s more. Apple ships Time Machine with every Mac. Time Machine is a backup program that runs in the background of your Mac and backs up your drive to either an external drive or a drive located on the network — whether that’s another Mac or a centrally located file server like a NAS.

The theory is that if you have one Mac, iCloud Drive, and Time Machine backing up to a server or spare external drive, you have a fully valid 3-2-1 backup strategy and your files are safe.

Except… not so much.

How MacOS represents iCloud ‘optimized’ files on disk

It’s important to understand this. 

If you fit the profile of someone with more storage in iCloud than on your local Mac, this is mission-critical stuff. It also took me quite a while to wrap my head around all the not-really-documented nuances. I’m going to start by showing you how various parts of this overall architecture work, and then I’ll put it all together and show you how it can fail.

Let’s start by taking a look at a typical directory managed by iCloud Drive. This is on one of my 256GB M1 Mac Minis.

If you look at the files indicated by (1), you can see the cloud download icon is shown next to all the screencasts (which are video files). If you look at (2), it appears that those screencasts consume anywhere from 6.4 to 42.4 megabytes (MB) of storage. The two template files indicated by (3) do not have the cloud download icon and show they consume 1.2MB of storage.

But now, let’s look at the same directory using Terminal:

There’s some interesting stuff to observe. First, at (1), you can see the screencasts are really only taking 186 bytes of storage on the local SSD. By contrast, the template files at (2) are actually consuming roughly 1.2MB on the drive.

Also: What is torrenting and how does it work?

But now, look at (3). Notice the dot in front of the word screencast. Dots in MacOS (and Linux) are used to denote hidden files. These are files not normally shown unless a special command is given (I used ls -la to show them). But as we saw earlier from the Finder screenshot, the screencasts are visible. That’s because of the .icloud extension shown at (4).

MacOS represents those file stubs with a dot at the beginning of the file name and .icloud at the end. Those four files are not my screencasts. They’re just tiny little files that represent my screencasts, which are living up in Apple’s iCloud data center.

You can see how there’s a lot of local storage savings. All four of those screencasts add up to 64.2MB of storage. But they’re only consuming 744 bytes, or roughly 1/85th of the storage they would otherwise be consuming.

What Time Machine actually backs up

When set up properly, Time Machine runs once an hour and backs up your Mac. Or so you’ve been told. In reality, Time Machine only backs up the full files that reside on your Mac. Because the .icloud files aren’t really on your local drive, Time Machine has nothing to “grab onto,” so it doesn’t back them up.

In the case of the directory shown above, Time Machine will back up the template files, but those hard-to-make screencasts were not preserved in the backup.

What Time Machine actually restores

Time Machine represents iCloud-optimized files in a couple of ways. In this image of Time Machine’s current view, you can see the four screencasts. They show backup sizes in megabytes (1), but they also show the iCloud download indicator (2).

It sure looks like Time Machine backed up the full file. But it didn’t. Here’s what happens if you try to do a restore:

Notice the file is brought back as, seemingly, an MP4 video. But also notice that it’s only 186 bytes (1). Also notice that if you try to run the restored file, assuming MacOS will go and get it from iCloud, you get an error (2).

But it gets worse. Let’s travel back in time a few days:

This is the Time Machine record for Jan. 14, exactly two weeks before I’m writing this. I haven’t been in that directory at all in months and months. Notice anything? Or, more precisely, notice anything missing? Yep, at (1) there should be four screencast files. They’re not there.

Yes, I see that hand straining in the back of the room…

“But, but, but, didn’t iCloud back up those files once, back when they were full files?”

The answer is… probably. But when? Are you going to go digging through every day to try to find those files? You might, but it’s a pain.

And, what if you want to recover an entire directory? Different files were stubs on different days. Are you going to go day by day, directory by directory and try to recover what may well be thousands of files, each with a different directory listing on a different day?

It’s just not practical.

So now that you understand how optimized files are represented in the local file system, and how they’re only sort of backed up and not really recoverable, let’s turn our attention back to iCloud Drive storage in the cloud.

iCloud limitations

Apple’s iCloud implementation has some startling limitations beyond the fact that there is no plan beyond 2TB. That’s a discussion for another article. In this article, I’m referring to architectural limitations, specifically:

1. You can’t specify that certain files or folders are not to be synced. It’s all of Documents and Desktop, or nothing.
2. You can’t easily tell if syncs are failing or the last time a sync occurred. You can view the status bar or compare folders on your Mac with the cloud, but there’s no “last time synced” or the ability to look at a sync log to really confirm everything’s working — other than looking at each file in each folder and comparing them.
3. You can’t tell iCloud to sync on demand. In other words, if iCloud hasn’t synced, you can’t just click a menu item to “Sync Now.”
4. The only way you can force a sync is to log out of iCloud entirely. This causes iCloud to remove all the iCloud data from your Mac (although you can ask to keep a copy of your data in a backup archive).
5. iCloud doesn’t provide notifications or email alerts telling you if syncing has stopped. In fact, iCloud doesn’t tell you anything about iCloud Drive other than overall space used.

That fourth point has some interesting and probably unintended consequences. If iCloud isn’t syncing, the only way you can get it to start up again is for the local data to be removed and the data in the cloud to be brought back down to your local machine.

This makes the files stored in the cloud the copy of record. Everything rebuilds based only on what’s already stored in the cloud. Worse, if you tell MacOS to save your old files before logging out, that archive is incomplete — containing only the files that were downloaded to the Mac at the time of the snapshot. 

Herein lies a serious problem. Apple always assumes the data in iCloud is the definitive copy of your data. Any actions you take to fix syncing problems involve bringing that assumed-to-be-definitive copy of data down from the cloud to your local machine.

But what if the cloud dataset isn’t correct?

That’s the interesting problem. Since you can’t force a sync, if iCloud fails, all you can do is download the snapshot that’s in the cloud. You can’t tell iCloud that you want to trigger a sync and compare the local drive with the cloud. It’s either when iCloud wants to do it, or when you log out and then log back in.

That, of course, involves clearing your local machine and bringing (possibly incomplete) data back down from the cloud.

Recommendations, or where we go from here

The obvious answer is IT best practice: Make a complete backup of all your files outside of iCloud. That way, if there’s an iCloud failure (which Reddit seems to think is all too common), you can recover from your backup.

But if your local drive is too small to host the entire iCloud Drive dataset, you can’t count on getting a complete backup.

That’s where my two interconnected recommendations come into play. 

First, get a Mac with a big enough drive to hold all your files. Second, turn off Optimize Mac Storage. This will bring down full copies of all your files.

I hesitate to make that first recommendation because it’s expensive. Apple charges a whopping $800 to provision a computer with 2TB instead of 256GB. This is particularly galling when you can buy a 2TB Samsung T7 SSD for $170.

I’m also making a complete about-face from my previous recommendations. For years, I’ve been recommending getting a relatively small amount of Mac storage and using an inexpensive external SSD. I mistakenly believed that Time Machine did full backups and trusted that you could store the rest in the cloud with Optimize Mac Storage. My change of heart is the result of my deep dive into the workings of Time Machine and Optimize Mac Storage, and the learnings documented in this article.

Yes, you can move Documents and Desktop (actually, your entire Home folder) to the SSD by going to Settings→Users & Groups, and then right-clicking on the user name and choosing Advanced Options.

But when you do, you get this very unnerving warning:

In very, very tiny print, it says: “Changing these settings might damage this account and prevent the user from logging in. You must restart the computer for the changes to these settings to take effect.”

Note the words I’ve bolded. Might damage the account. Might prevent the user from logging in. We’re doing all this to prevent data loss, not send the Mac into spasms of unrest.

So, to be safe, I’m now recommending that you have at least one machine with enough local storage to contain full copies of all your files, which you can then back up using a mechanism other than Time Machine (I use ChronoSync to the Synology NAS). This machine needs to be equipped with at least 2TB internal storage and have Optimize Mac Storage turned off.

It’s an expensive option, but losing your data to Optimize Mac Storage’s completely non-optimal storage architecture can be far more expensive.

---

You can follow my day-to-day project updates on social media. Be sure to follow me on Twitter at @DavidGewirtz, on Facebook at Facebook.com/DavidGewirtz, on Instagram at Instagram.com/DavidGewirtz, and on YouTube at YouTube.com/DavidGewirtzTV.

READ MORE HERE