Thursday, January 19, 2012

Software deployment is no rocket science

Image: dream designs /
Technically speaking, software deployment requires little more than knowledge of the application's installation or update procedures, shell scripting and systems administration skills. So what's all this fuss about "the complexity of software deployment" that we can read about?

A financial software that processes millions of transactions per hour or a mobile network monitoring system that tracks the quality of every single call for an entire country in real-time, now these are example of complex systems which raise serious challenges.

Updating software involve copying files, setting up application and system security parameters, upgrading database schema and possibly configuring startup/cron configurations; that's about it.

So where is this so called complexity? And why is it that, more often than not, it takes less time to develop, test and package a new software update than to upgrade a customer deployment?

Ok, I admit that I'm leaving out some details but the truth is that while it doesn't take a rocket scientist to do software deployment, there are difficulties of a different kind:
  • It's a mind-bogglingly dull task: repetitiveness is an attention killer and increased release frequency resulting from agile development processes doesn't help.
  • Some applications must be updated on tens of servers. This type of repetition has even worse consequences on the mental sanity of the deployers.
  • Previous application configuration parameters must be re-applied to the new update. Since these are not always strictly documented, the safer procedure usually involves a careful and time-consuming inspection of the differences between old and new versions of all the files to copy changes over.
  • Implicit knowledge: the documentation of installation procedures is often slim and riddled with unwritten assumptions, relying on the experience gathered by deployment engineers over time.
All this introduces serious operational risks in the deployment process and so require additional time for double-checking and then trouble-shooting the new setup. Even then, a large proportion of defects, often insidious, that surface at production are directly attributable to simple mistakes during configuration.

So don't go looking for rocket scientists. All you need is a bunch of mindless and unerring drudges or ... empower your IT team with good agile deployment software to make their work immensely more efficient and rewarding but that will be another story.


Anonymous said...
This comment has been removed by a blog administrator.
Willie Wheeler said...

It's not rocket science, but even once you replace the drudgery of manual deployments with deployment automation, there is an entirely new set of challenges that awaits.

A proper deployment process is part of a larger configuration management practice. Many organizations--even those using deployment automation--struggle with issues of configuration drift, mismatch between deployment and operational tools (e.g. apps that appear in one tool must somehow be kept in sync with apps that appear in others), inability to verify DR plans on a continuing basis, difficulties with HA deployments (both zero-downtime and multiple data center), alignment with change management, blah blah blah.

Solving these issues involves overcoming some challenging technical hurdles; e.g., defining a CM schema that imposes enough constraints to prevent configuration drift while being flexible enough to be usable across hundreds of applications.

But perhaps even more challenging, at least in larger workplaces, are the organizational difficulties that arise. There are religious wars about tools, there are managers who think that the fixes are primarily around people being more disciplined, motivated and perfect than they actually are, there are rogue sysadmins who make changes that the deployment automation doesn't see, etc.

kwatee said...

@Willie: I fully agree with your comments. The larger workplaces IT organizations may be more mature in the adoption of datacente automation tools but software alone is not a silver bullet because of the issues you mention which will still have to be dealt with even though a number of tools can deal with HA and "rogue modifications".

On the other hand, the number of smaller organizations or enterprise software vendors that painstakingly deploy updates of business critical applications manually is simply flabbergasting. These guys are still in the stone age mostly because they see the "pro" solutions as being expensive (cost and/or learning curve) and overkill for their needs.

itoctopus said...

Hi Kwatee,

I think you are overseeing the fact that not many people know to think as developers. If you think about it, we do think differently than others. We know how to approach things when they are broken, we know what to look for and how to look for things.

Not rocket science, I agree - but complicated enough for others to be perceived as rocket scientist.s

kwatee said...

@itoctopus: guilty as charged on the count of thinking as a developer. And you are right, the way that this issue is tackled by many people can make it seem rocket science simply because of the the myriad of nitty gritty details that must be attended to.

Non-developers will embrace the right tools when they remove the pain point without creating new ones (i.e. easy to learn and use) by keeping tedious and fastidious operations under the hood and letting them focus on the real issues that require their problem solving abilities.

Adoption Agency said...

Cheers for a very good post. You have given me satisfaction to every post i read here. Hope you keep on sharing relevant information to all your readers.

Automated Software Deployment said...

Seems like I'm a little late for the party but just the same I appreciate the almost spontaneous back and forth comments and replies. Potentially speaking software deployment can be no rocket science for the initiated but for most average I.T. organizations, automation and synchronization can be a daunting task and yes it does present new challenges.

Post a Comment