Verify, trust, then verify again

OR: the tale of the backup that wasn’t,

Also known as “clouds can be cloudy”…

a mini-review of iCloud backup for iOS devices…and some useful ‘how to’ information.

Background: As a side business, I do some small business consulting on IT services.  (I’m mostly a software development executive consultant, but People Ask For Help, right?)

I made an error with a client’s iOS data this week, where some non-synchronized data was lost.  One little button push…and now I’m on the hook for recovering the lost data.  (The bulk of the data is good, but the non-synchronized part from a device is lost, as iCloud backup for calendar data assumes that ‘normal synchronization’ is working. ARGHH.)

My client has to fix the problem (manually). This involves re-doing meeting invitations, and is NOT A GOOD THING.  Even if iOS was synching correctly, its backups have some serious limitations…see below.

I am NOT happy with this additional pain. That said…Always remember, it’s easier (or even just possible) to fix things when you follow the IT Pro Mantra:

  1. BACKUP first.
  2. Confirm the backup.  If it’s not tested, it doesn’t work! Make sure that everything you think is backed up, IS backed up.
  3. If the step isn’t reversible, see steps 1 and 2.
  4. If you can’t backup, and the step isn’t reversible, have a second person double-check your work before pulling the trigger.  And reconsider if it’s the right thing, and what you are going to do when (ha ha, ‘if’) it goes wrong.  There are too many systems where it is very very easy to do something irreversible, because that “shouldn’t happen.”
  5. For people: Trust, but verify.  For automated systems, verify, then trust, then verify again.

So, a ‘micro review’ of iCloud backup for iOS devices:  only barely acceptable.  Better than nothing, but you still have to check and verify, and realize that there are serious limitations.  Computer-based backup in iTunes is better, in that it restores without interpreting or synching data, so *non-synchronized data is preserved*, but ONLY if the backup is made while iCloud synching is turned off (a screenshot from my system to preserve client confidentiality):

Screenshot 2016-02-03 11.12.58.png

iCloud backup *loses non-synchronized data*. Even locally synched iTunes backups will lost non-synchronized data, so BE CAREFUL when relying on iTunes backups.

iCloud supports backups for:

Files, Contacts, Calendars and Reminders, and Bookmarks.

Screenshot 2016-02-03 11.01.07

(Bottom of the screen,

Let’s walk through these:

  1. Files.  These are deleted files from iCloud Drive.  Fairly minimal.  See above
  2. Contacts.  These are snapshots of your SYNCHED contacts.  The bad news:  NOT your un-synched contacts.  The good news: if you restore, then there’s a backup made of your ‘current’ synched contacts.Screenshot 2016-02-03 11.07.54
  3. Calendars.  Same caveats as contacts (only synched), but worse.  Sorry folks.  ALL sharing information is removed.  ALL scheduled events will be cancelled and recreated, and invitations re-issued.  And as above, the archive will replace all your calendars and reminders on all devices.
  4. Bookmarks:  same as above.

Here’s my specific recommendation:  If you are fixing a problematic non-synching iOS device, you have two choices.

  • Lose all of the data that is only on the device by turning off the local copy and restoring from iCloud.  OK if the device wasn’t used to enter data.  OR…
  • If you really need the unsynched data:
    1. turn off ALL iCloud synching, and do NOT ‘delete the local calendar.’  Oops.
    2. do a complete backup of the device, turning ON iTunes synching of calendars and contacts (again, by turning OFF iCloud synching on the device).  Confirm that this is set correctly in iTunes: [THIS]Screenshot 2016-02-03 12.01.47not [THAT]. Screenshot 2016-02-03 11.12.58
    3. restore to a scratch device if you possibly can,
    4. inspect and verify the contents.
    5. Manually inspect all non-synchronized content,
    6. record it, and
    7. then wipe and restore the synched content from iCloud.
    8. THEN manually restore the non-synchronized content.
    9. Keep in mind that IF you ever need the ‘archived’ synch content, you WILL lose the current content unless you restore ONLY to a non-synched scratch device, so that archive isn’t useful for very long.

That ends the ‘how to’ part of this post.  Next: editorial content.

I have to say that we, as a product industry, do not do a very good job on backups, restores, and synching. Apple, Google/Android, Microsoft, Oracle and RIM are definitely dropping the ball by only supplying audit/log/reconstruction tools to enterprise customers. Individuals and small businesses deserve better than “Oh well, that’s how cloud and mobile play together.”

Unfortunately, this isn’t new; over my career, I’ve dealt with tape backups where databases are only ‘crash consistent’ (that is, operationally unusable without major work), so the assumption that “oh, we can just restore from backup if this doesn’t work” wasn’t viable because it was not tested.  Going back further, I recall a Unix backup (SunOS) where all that was on the backup tape was “file not found” since a symbolic link wasn’t followed during the backup process…and the restore had never been tested.  (Fortunately, we had separate full file backups; remember the IT Pro Mantra!)

When I was with EVault (now a part of Carbonite; hi folks!), it was shocking how many of our competitors essentially only did backups, NOT working restores.  In essence, these competitors were betting that backups were only a check-box item, and if a disaster occurred, a big download or physical media delivery was sufficient…and actual working systems were the customer’s problem.

Does your backup actually restore?  Are you sure?




How Rogue Techies Armed the Predator, Almost Stopped 911, and Accidentally Invented Remote War

How Rogue Techies Armed the Predator, Almost Stopped 911, and Accidentally Invented Remote War:

On the afternoon of October 7, 2001, the first day of the war in Afghanistan, an Air Force pilot named Scott Swanson made history while sitting in a captain’s chair designed for an RV. His contribution to posterity was to kill someone in a completely novel way.


Commentary (Dak):  Ah, the power of a motivated team focused on solving problems rather than just serving the huge organizational process gods.  Those processes have value in terms of scale and defensibility of acquisitions for well understood technologies (K-rations, anyone?) but they have their downsides.

Why morale matters

The McKinsey Quarterly has a good interview with Brad Bird (director of The Iron Giant, The Incredibles, and Ratatouille). I’ve been following Pixar with great interest over the years, partly because I have a couple of old friends who work there (Hi guys!), and partly because I really believe in what they are doing (changing from being a software vendor in a niche market to being a major motion picture studio: brilliant!). In my opinion, Pixar is the poster child for the “eat your own dog food” school of management, and deserves their success. (How good is Renderman? Well, it’s good enough that we’ve won Oscars with movies we’ve built on it!)

In my experience, THE key issue on the performance of teams is to get the morale and the synergy of the teams going. This involves selecting the right people, keeping the great players in the team, and keeping the ideas flowing.

Here’s a great quote from the interview:

The Quarterly: It sounds like you spend a fair amount of time thinking about the morale of your teams.

Brad Bird: In my experience, the thing that has the most significant impact on a movie’s budget—but never shows up in a budget—is morale. If you have low morale, for every $1 you spend, you get about 25 cents of value. If you have high morale, for every $1 you spend, you get about $3 of value. Companies should pay much more attention to morale.

Brad is being very low-key here; the emphasis is mine. In my experience, this is exactly correct.

Read the rest of the interview for how and why Brad worked on morale.

What are you doing to increase the morale of your team (and your family, and the broader group of people you work with) TODAY? I’m talking about hugs and compliments; what are you doing to recognize people as individuals, to listen to them, and to make them feel listened to?

More later in the blog, on building a team of “Developers versus Programmers.”

Have a great weekend,

Announcing Dakworks, and doing today’s "idea warmup"

Dakworks is an information technology, software development, editing, and management consultancy. As the principal consultant, I believe that systems and software must be built to make great things possible, to generate new possibilities, and to liberate people from onerous and repetitive tasks.

At present, the purpose of this blog will be occasional commentary, opinion, and links to “things that I found interesting” in the spirit of the new, trendspotting, and nifty uses of leading and trailing edge technologies.

Today’s cool toy spotting (Thanks to Malcolm Stanley at Strategic Thinking and Execution for pointing it out):

This uses the Arduino controller and an add-on sensor package to monitor plant status and post the results on Twitter.

The application itself is silly, but think about the implications: Twitter can serve as a personal mash-up dashboard for your home, your car, your pets, your pre-writing kids, and not just your blogosphere and mobile device friends.

For example, imagine an outdoor motion sensor and camera *as a Twitter feed*. Outdoor motion sensor goes off, not worth an alarm…but if you get a call from your alarm company, AND the outdoor motion sensor went off, you’d check your camera and confirm that it’s likely NOT a false alarm. Twitter’s ongoing ‘low quality’ information stream is a great way to moderate and present this data, since it’s redundant, and truly a ‘value add’ rather than a primary source of data.

The key here is to put people in the loop when interpretation is required. This is exactly what Dakworks is all about.

Thanks for reading,