Software projects are late and over-budget, yet still fail to meet all requirements and are buggy. What is the problem with software engineering processes and tools that lead to such terrible results? Nothing. Most complex projects are late and over-budget, and still fail to meet all requirements and are buggy. Boeing is paying sizable penalties to customers because their latest plane is late and over-budget. Most defense contracts are late and over-budget. In fact, it’s rare to find a contractor that can fix your house without being late and over-budget. These problems are not unique to software; therefore, software isn’t the problem.
The problem is entirely people. The person managing the task is forced to guess when the task will be complete (when will you cure cancer?). The guess inevitably becomes a hard deadline. The task is ill-defined and vague (make the game more fun). The employees may not have the right skills to do the job effectively. Add to this the general dysfunction of any bureaucracy and there is very little hope of accomplishing anything on time. Shuffling people into different roles and slapping on new processes and tools will do nothing to solve the core problem: most people and organizations are not capable of delivering results efficiently.
Should you throw your hands up in despair? Yes. Most IT organizations are asking too much of their staff. It’s like asking a nurse to perform surgery. If management lowers their expectations, they can get good, consistent results from their IT staff. Otherwise, they are doomed to failure no matter what software engineering bandwagon they jump on. If companies with great talent and tools and experience (MS, Sun, IBM, Google, etc.) still have significant problems building software, how can anyone else do a good job? Stop trying and you’ll be much happier.