The Centralized vs. Distributed Debate Continues

After my previous post, “Can Globally Distributed Development Really be Supported with a Central Subversion Server? ” CollabNet’s Jack Repenning recently responded with a post of his own: “Optimizing Globally Distributed Source Management”.

Needless to say, everyone at WANdisco has tremendous respect for Subversion and what the Subversion community has achieved. We’ve built a significant portion of our business around Subversion because of that respect. My post had everything to do with implementation architecture and nothing to do with the quality of Subversion. In fact, it could have been taken and applied to a debate over whether or not to centralize or distribute a database, or any other application.

And while Jack and I still agree on the list of tradeoffs in the centralized versus distributed debate, including his additional item of enterprise workflow, we still see different outcomes:


WAN costs, in terms of bandwidth and latency in large corporations are not merely a problem of, “…configured “Quality of Service” policies colliding with evolving use patterns which can be reconfigured…” WANdisco’s Fortune Global 1000 customers moved away from relying on a central Subversion server, even though they did control their networks and had ample resources to obtain extra bandwidth. Some of them are even network service providers. The bottom line is that no matter how good the network is, there’s still the unavoidable speed of light problem, which gets compounded by the number of WAN round trips required to complete a transaction. Then there’s always the risk of timeouts with each network hop as data gets transmitted over a WAN. There’s no forward recovery after a timeout, so transactions have to be resubmitted and data can be lost. Finally, if remote users lose their connection to the central server, they have no repository access at all.

Server performance can be addressed up to a point by adding more memory, CPU or disk. Some of WANdisco’s customers, including those that actually manufacture servers, did just that, or asked their hosted service provider do it for them as a stopgap. But no increase in the capacity of a central server can get away from the fact that it’s a single point of failure that’s destined to become a performance bottleneck again. Distributing Subversion repositories with the right solution gives IT organizations the ability to balance workload across their development sites and locate data close to users, which by default leads to better performance, scalability and availability (the next item on the list) in a way that no central server implementation can.

Availability is ultimately a function of the fail-over and disaster recovery features of an implementation, and since there must be something to restore from, the backup solution. Most large data centers rely on disk mirroring, coupled with backups taken at regular intervals in case the mirrors fail, to provide the data to restore from. There are three challenges with disk mirroring solutions: (1) they’re designed primarily to protect against disk failure, not total server failure, so the application still has to be brought up on another server, (2) some manual intervention is required so the risk of human error leading to data loss and extended downtime is real and (3) they only work over distances covered by a metro-LAN at best, not a WAN.

Continuous hot backup and automated disaster recovery are included with Subversion MultiSite, so there’s no need to rely on disk mirroring solutions and incremental backups, or manual procedures of any kind. Servers can even be taken offline for routine maintenance without interrupting user access, making full 24 by 7 operation a reality in a global environment. As soon as a server comes back online it catches up automatically with the other servers.

Enterprise work flow in this context refers to the ability to map and selectively replicate groups of Subversion repositories to only those sites where the development teams that use those repositories are located. As projects, teams and locations shift and change the replication map needs to change with them.

While this can get complicated without the right tools, large enterprises have implemented Subversion MultiSite partly on the basis of its ability to handle selective replication. They didn’t see it as a problem when they moved away from a central server.
One of the key features provided by Subversion MultiSite is the ability to administer all sites from a single location. Selective replication is just one facet of this.