Introduction
From 2001 to 2004 my former employer wrestled with offshore development and cost-cutting;
a series of development projects were moved to a vendor in Pune, India and the US development teams were let go.
While the loss of friends and talented co-workers was painful, it was only the beginning as
over time it became clear that costs were not reduced nearly as much as predicted,
that delivery timetables remained unpredictable, and that delivered software quality was suspect.
After many conversations on the issues of offshoring with a variety of managers and developers some core issues have emerged that require broader discussion than this web page!
This essay contains a realistic example of why offshore development time is increased due to communication issues, a short discussion of why offshore problems are rarely covered in the media, and a couple of ideas for how to cope with this new way of building software...
Estimating Soft Costs for Offshoring Software Development
The real face of outsourcing - is soft costs and delays. I worked on four outsourced projects over three years with my previous company and most of the experience was grim. Frankly, cost savings are overblown when you look at the communications overhead and code re-work needed when communications are not perfect...
Average cost for a single change to workflow in code, measured in person-hours
Example: Typical Feature Iteration
Task | Local Team | Offshore | Why are they different? |
Discuss Requirement | 4 hr | 8 hr | Video conference setup, invites, time change |
Write Detailed Requirement | 4 hr | 24 hr | Requirements must be more detailed due to communication barriers and cultural issues - developers will not raise conflicts in docs, but will build to spec, then make changes once errors are found. American programmers will generally rip you a new one if your docs are contradictory, or just fix the issue as they see fit. |
Handoff Meeting with Development | 2 hr | 12 hr | Video conference setup, invites, time change, more offshore developers for same tasks |
Follow-up Question from Development | 2 hr | 12 hr | |
Mockup Design for Feature - UX Model | 8 hr | 8 hr | |
Test proposed design, refine requirement | 8 hr | 0 hr | US office won't see mockup due to time contraints, technical delivery and install issues, server access, etc., etc. |
Approval Meeting | 4 hr | 8 hr | Video conference setup, time change. Plus, since there was no testing of design, this approval meeting is the only chance to catch errors and adjust requirements |
Write Code for Feature | 24 hr | 24 hr | |
Self-Test Code | 8 hr | 8 hr | |
Team Code Review | 4 hr | 0 hr | Not in CMM... |
QA Test Specification | 4 hr | 12 hr | More detail needed for offshore specs |
QA Test Spec Approval | 4 hr | 12 hr | Video conference setup, invites, time change |
QA Testing | 8 hr | 16 hr | More, but cheaper, bodies |
Release Documentation | 8 hr | 12 hr | |
Review Documentation | 4 hr | 12 hr | Video conference setup, invites, time change |
Revise Documentation | 2 hr | 12 hr | |
Refine Help | 4 hr | 8 hr | |
Refine Help Again | 0 hr | 16 hr | Communication issues |
Production Coordination/Handoff | 8 hr | 8 hr | |
Support for Changes | 4 hr | 12 hr | |
Total | 114 hr | 232 hr |
So, if it takes twice as much work, and the code is not of the same quality, but you only paid half the hourly rate - is it a bargain?
Other Effects
There is a aphorism that nature abhors a vacuum
...and economies abhor differences in costs
This is the natural law of economics that underpins supply and demand - if there is a disparity in the cost of scarce resources then supply and demand quickly reaches a new equilibrium. Capitalism exploits differences in cost v. worth to make money, but over time the cost of any resource is adjusted by demand.
Leaving aside the issue of where they live, talented programmers are an extremely limited resource and probably will continue to be so until we can develop some sort of machine intelligence (and at that point, we may well all be out of a job).
The truth is after more than forty years of software design methodology development and countless consultants and development tools, software development is still somewhere between art and craft. There are dramatic differences between individual programmers that make some of them worth much more than others.
Example: Salaries for qualified programmers in India have been rising at double-digit percentage for more than five years.
Example: Many outsourcing firms now hire recent graduates at lower cost and train them on your projects - once these graduates have 6-12 months experience in your cool, leading-edge technology they will leave the initial vendor for higher paying jobs since the outsourcing contracts do not allow the shops to give huge raises and maintain their economic profit. This is the opposite of what complex software development requires - experienced and talented programmers often take weeks (or months) to fully understand large complex software projects in order to contribute effectively, while unseasoned programmers who start coding right away inevitably do damage, or at least create code that will have to be 're-factored' later.
Example: Turnover is so high in most outsourcing shops that they have adopted strange coping behaviors - such as one shop I dealt with that kept 20 freshly hired programmers on 'the bench' looking over the shoulders of the real developers and preparing to step in when they left for another job.
Example: The same outsourcing shop was beginning to outsource their own work in turn to offshore programmers in China.
OK, so why don't we hear this side of the story in the media?
Modern 'flat' hierarchical organizations do not lend themselves to thoughtful understanding of real project issues - the pervasive trend to summarize issues for effective time usage masks many complex problems from management. Terms like 'human capital' are used to de-personalize and package management concepts into insulated and compartmentalized decisions.
Many development managers clearly understand the soft cost issues of communication and quality in the outsourcing trend, but their management in turn are increasingly from other fields and do not have a clear understanding of software development. With slow-developing trends on costs and management churn the numbers may be clouded for years...
- Many late-adopter companies are only now committing to outsourcing
- It takes at least one full development cycle to understand the true costs and timetables for outsourcing, and usually the first project overrun is dimissed as initial learning costs, that means that two cycles are needed to see the trend. With management turnover on both ends of modern development organizations valuable trend information can be lost or ignored
- Managers at outsourcing vendors are smart and savvy - they have a variety of explanations for the ever-rising costs and timetables on projects and, as they will often point out, they are still 1/3 to 1/2 the hourly cost of American developers
- Business leaders are still smarting from the leverage that american programmers had over various industries in the 1990's and are very interested in getting development costs back under control. The threat of outsourcing is a very effective tool in managing development budgets and salary demands
- Obsession with other parts of the business, poor communication
- Modern project management tools such as MSProject give the illusion of competence and control without reality - Humans finish what you measure. If you measure tasks, they will complete tasks, but tasks do not equal good software.
- A lot of senior business leaders have bet their bonuses (and their political fortunes within their companies) on the savings from outsourcing, or have gotten good press coverage from the outsourcing story - to admit costs are equivalent or to reverse positions now is difficult
What Can We Do?
This is a trend that seems unstoppable; we need to find better processes so we can get better results. Communicate. Discuss. Find good examples of companies doing outsourcing correctly, or of companies abandoning outsourcing, and present them to your management. An open and frank discussion that uses real-world examples will go a long way to help management without technical backgrounds to understand the challenges of this style of business.
The most critical part of this methodology is the requirements specification - since there are time and cultural barriers that prevent the constant back-and-forth discussion of design intent that is inherent in local software development, a quality specification becomes the only practical channel for the software built. So be careful what you ask for...
- Be crystal clear in your requirements and specifications
Do as much testing with design mockups as possible to get a polished design before you write the final specs - Write specifications with user Goals and with detailed workflow examples
Do not assume that the offshore team understands existing processes, they probably do not even have a working version of your existing application - Review the specs with *everyone* to validate the requirements and to set expectations
- Ask for code reviews, and participate
Specify code reviews and feature reviews *before* milestone deliveries in the contract; this gives you a chance to catch serious errors before code delivery and helps you understand their development skills and weaknesses - Make sure that developers that write bad code are fixing their own mistakes
And applying fixes across all the code areas they touched - and that they are cross-training all the offshore team to not reuse old, bad code