backups human-in-the-loop

Verify, trust, then verify again

OR: the tale of the backup that wasn’t,

Also known as “clouds can be cloudy”…

a mini-review of iCloud backup for iOS devices…and some useful ‘how to’ information.

Background: As a side business, I do some small business consulting on IT services.  (I’m mostly a software development executive consultant, but People Ask For Help, right?)

I made an error with a client’s iOS data this week, where some non-synchronized data was lost.  One little button push…and now I’m on the hook for recovering the lost data.  (The bulk of the data is good, but the non-synchronized part from a device is lost, as iCloud backup for calendar data assumes that ‘normal synchronization’ is working. ARGHH.)

My client has to fix the problem (manually). This involves re-doing meeting invitations, and is NOT A GOOD THING.  Even if iOS was synching correctly, its backups have some serious limitations…see below.

I am NOT happy with this additional pain. That said…Always remember, it’s easier (or even just possible) to fix things when you follow the IT Pro Mantra:

  1. BACKUP first.
  2. Confirm the backup.  If it’s not tested, it doesn’t work! Make sure that everything you think is backed up, IS backed up.
  3. If the step isn’t reversible, see steps 1 and 2.
  4. If you can’t backup, and the step isn’t reversible, have a second person double-check your work before pulling the trigger.  And reconsider if it’s the right thing, and what you are going to do when (ha ha, ‘if’) it goes wrong.  There are too many systems where it is very very easy to do something irreversible, because that “shouldn’t happen.”
  5. For people: Trust, but verify.  For automated systems, verify, then trust, then verify again.

So, a ‘micro review’ of iCloud backup for iOS devices:  only barely acceptable.  Better than nothing, but you still have to check and verify, and realize that there are serious limitations.  Computer-based backup in iTunes is better, in that it restores without interpreting or synching data, so *non-synchronized data is preserved*, but ONLY if the backup is made while iCloud synching is turned off (a screenshot from my system to preserve client confidentiality):

Screenshot 2016-02-03 11.12.58.png

iCloud backup *loses non-synchronized data*. Even locally synched iTunes backups will lost non-synchronized data, so BE CAREFUL when relying on iTunes backups.

iCloud supports backups for:

Files, Contacts, Calendars and Reminders, and Bookmarks.

Screenshot 2016-02-03 11.01.07

(Bottom of the screen,

Let’s walk through these:

  1. Files.  These are deleted files from iCloud Drive.  Fairly minimal.  See above
  2. Contacts.  These are snapshots of your SYNCHED contacts.  The bad news:  NOT your un-synched contacts.  The good news: if you restore, then there’s a backup made of your ‘current’ synched contacts.Screenshot 2016-02-03 11.07.54
  3. Calendars.  Same caveats as contacts (only synched), but worse.  Sorry folks.  ALL sharing information is removed.  ALL scheduled events will be cancelled and recreated, and invitations re-issued.  And as above, the archive will replace all your calendars and reminders on all devices.
  4. Bookmarks:  same as above.

Here’s my specific recommendation:  If you are fixing a problematic non-synching iOS device, you have two choices.

  • Lose all of the data that is only on the device by turning off the local copy and restoring from iCloud.  OK if the device wasn’t used to enter data.  OR…
  • If you really need the unsynched data:
    1. turn off ALL iCloud synching, and do NOT ‘delete the local calendar.’  Oops.
    2. do a complete backup of the device, turning ON iTunes synching of calendars and contacts (again, by turning OFF iCloud synching on the device).  Confirm that this is set correctly in iTunes: [THIS]Screenshot 2016-02-03 12.01.47not [THAT]. Screenshot 2016-02-03 11.12.58
    3. restore to a scratch device if you possibly can,
    4. inspect and verify the contents.
    5. Manually inspect all non-synchronized content,
    6. record it, and
    7. then wipe and restore the synched content from iCloud.
    8. THEN manually restore the non-synchronized content.
    9. Keep in mind that IF you ever need the ‘archived’ synch content, you WILL lose the current content unless you restore ONLY to a non-synched scratch device, so that archive isn’t useful for very long.

That ends the ‘how to’ part of this post.  Next: editorial content.

I have to say that we, as a product industry, do not do a very good job on backups, restores, and synching. Apple, Google/Android, Microsoft, Oracle and RIM are definitely dropping the ball by only supplying audit/log/reconstruction tools to enterprise customers. Individuals and small businesses deserve better than “Oh well, that’s how cloud and mobile play together.”

Unfortunately, this isn’t new; over my career, I’ve dealt with tape backups where databases are only ‘crash consistent’ (that is, operationally unusable without major work), so the assumption that “oh, we can just restore from backup if this doesn’t work” wasn’t viable because it was not tested.  Going back further, I recall a Unix backup (SunOS) where all that was on the backup tape was “file not found” since a symbolic link wasn’t followed during the backup process…and the restore had never been tested.  (Fortunately, we had separate full file backups; remember the IT Pro Mantra!)

When I was with EVault (now a part of Carbonite; hi folks!), it was shocking how many of our competitors essentially only did backups, NOT working restores.  In essence, these competitors were betting that backups were only a check-box item, and if a disaster occurred, a big download or physical media delivery was sufficient…and actual working systems were the customer’s problem.

Does your backup actually restore?  Are you sure?



Dak human-in-the-loop inspiration linkedin people time management Worth reading

Why morale matters

The McKinsey Quarterly has a good interview with Brad Bird (director of The Iron Giant, The Incredibles, and Ratatouille). I’ve been following Pixar with great interest over the years, partly because I have a couple of old friends who work there (Hi guys!), and partly because I really believe in what they are doing (changing from being a software vendor in a niche market to being a major motion picture studio: brilliant!). In my opinion, Pixar is the poster child for the “eat your own dog food” school of management, and deserves their success. (How good is Renderman? Well, it’s good enough that we’ve won Oscars with movies we’ve built on it!)

In my experience, THE key issue on the performance of teams is to get the morale and the synergy of the teams going. This involves selecting the right people, keeping the great players in the team, and keeping the ideas flowing.

Here’s a great quote from the interview:

The Quarterly: It sounds like you spend a fair amount of time thinking about the morale of your teams.

Brad Bird: In my experience, the thing that has the most significant impact on a movie’s budget—but never shows up in a budget—is morale. If you have low morale, for every $1 you spend, you get about 25 cents of value. If you have high morale, for every $1 you spend, you get about $3 of value. Companies should pay much more attention to morale.

Brad is being very low-key here; the emphasis is mine. In my experience, this is exactly correct.

Read the rest of the interview for how and why Brad worked on morale.

What are you doing to increase the morale of your team (and your family, and the broader group of people you work with) TODAY? I’m talking about hugs and compliments; what are you doing to recognize people as individuals, to listen to them, and to make them feel listened to?

More later in the blog, on building a team of “Developers versus Programmers.”

Have a great weekend,