Synchronizing the Cloud

[time]

As a consequence of my misery with Microsoft's blighted Offline Files, I've greatly accelerated my research into online storage and using 'The Cloud' to synchronize geographically divergent devices, eg. laptops and phones.

Purist cloud is, IMO, stupid and impractical. It all comes down to using some form of local caching at different levels to overcome the practical limitations of connectivity.

My idea is, very loosely, to publish a workgroup's document folders to the cloud and subscribe laptops and phones to specific folders to allow 2-way/multi-way synchronization. So in my view of the world, there should always be at least two complete copies, the workgroup server and the cloud, with cached copies on other devices as appropriate.

So far, I've looked at about 20 products from companies that really should have stuck to selling snake oil. I pulled the trigger on one called "ZumoDrive". To test it, I linked a desktop folder of photos to the cloud, then accessed it from the ZumoDrive app I installed on my Android phone.

Upload was slow (50KB/s, possibly due to the cloud being in the US and me in Oz), but I was able to both browse thumbnails of the photos and then step through them on the phone. Cool so far. Except. One of the 12 photos didn't finish loading; in its place the phone displayed an animated 'wait' icon.

Same result when I used a browser to check the cloud. Trouble is, ZumoDrive at both ends assured me that everything was okay. Anyway, I left it overnight and poked around with it this morning, but couldn't change the state of affairs, so I unlinked the original folder and relinked it, thinking that this would trigger a resync.

Imagine my horror as I watched my photos blink out of existence, one at a time ...

I managed to stop it after just 5. Later, I discovered them in the Recycled Bin, so it was an explicit Delete operation, not vastly accelerated bit rot. Needless to say, I've uninstalled ZumoDrive.

Would other people care to reveal their experiences?

 

[Mercutio]

Have you considered implementing WebDAV? Novell has a client that's quite nice, essentially allowing Windows users to treat files made available in a web server as if they were shared through SMB.

 

[Chewy509]

If you're a Windows only shop, and I shudder as I say this... but have you tried Sharepoint?

It has decent integration with Office applications, and you can host it all you self, and there is a very large 3rd-party add-on community.

But if you want an FOSS solution (that works like Sharepoint), there are few out there, like Sun Microsystems Glassfish (now branded slightly differently due to Oracle), and Hippo.

But as a tiered system like you are looking for, I haven't seen one that works as advertised.

PS. http://en.wikipedia.org/wiki/Enterprise_portal for more systems.

 

[timwhit]

Glassfish is an enterprise Java application server, I don't see how it would be useful for what Time is asking to do.

 

[Chewy509]

Re: Glassfish, I was thinking of WebDAV plugin for file management.

 

[Handruin]

I hesitate to recommend something from where I work, but have you heard of the EMC Atmos solution for distributed unstructured content? you can interface with it using their own RESTful interface.

 

[time]

WebDAV is just an interface, and quite a low level one at that.

You can go and hire storage direct from storage providers such as Amazon's S3. There are various low-cost tools to access this via WebDAV. They expose features such as auto-renaming new versions of a file with the date and time. That's lovely, but a bit 1990's in concept.

Handruin's solution at least mentions versioning, but it still looks like a lot of work to get to an end-user solution. There really seems to be a huge gap in the market here. Companies like Box.net, who have been around for years, have only just realized that it's not actually viable to access all your files exclusively through the Internet. Well doh!

Versioning is vital - without it, the cloud isn't a backup at all, even with synced copies. If I delete or overwrite a file on a single device, the change will be replicated everywhere, and I won't have an earlier version to go back to.

All the realistic solutions I looked at rely on added value from their own software and servers to deliver the necessary framework.

Every workgroup has common files that users need to access. Ideally, you don't want them duplicated across every workstation, but browsing them in the cloud may well be too slow for everyday use. This is why we have local file servers. My current but temporary solution can successfully mirror a local server to the cloud (and vice versa). When a user untethers, they can still access the common files through the cloud, albeit much more slowly.

Dropbox is a really popular solution, although that's got a lot to do with the fact that it's free as well as super-simple. It does actually have a couple of cool features, such as LAN sync (it's smart enough to realize that it can sync across a LAN segment rather than having to download for the cloud) and support for iPhone, Android and Linux. But it's far too primitive to be suitable for any other than individual users.

Spideroak does not have any limitations on which folders can be mirrored (eg removable or network drive) and doesn't create lock conflicts, so I've been running it on a Windows PC to sync Samba shares with the cloud. The company makes much of the fact that your data is encrypted with your password and that they cannot even see your file structure. Here's the thing (as I understand it): that means you can't change it without reloading all your data. So when a laptop is stolen, the only way you can protect your data is to trash your data stores and start again ...

Clearly, some form of user-based access control is essential. Scratch the two products I just mentioned.

You also need to be able to differentiate ReadOnly and ReadWrite, and importantly, which folders should be synced to laptops/phones and which should just remain in the cloud.

I don't think there's a single product that can even manage that short list of requirements, let alone also have native apps for iPhone, Android, Blackberry and Symbian. SugarSync does come close.

Most services don't support mobile devices (phones) at all. Some provide a mini-browser interface. But if you want the option of editing documents or uploading photos, you're more likely looking at native apps. Speed and usability are in a different league. A few of the providers are belatedly but frantically developing them, usually for iPhone/iPad first.

Then there's the issue of which storage provider to use: S3, Rackspace, Google, proprietary, etc. When I started this little adventure, I assumed it would be Google. If you use Syncplicity or Memeo - or Box apparently - it can indeed be. Memeo actually uses the previously mythical "G Drive" term. (Syncplicity has nice web admin, BTW).