Business & Technology Nexus

Dave Stephens on technology and business trends

Software Development Practices Must Change

with 4 comments

I think it might have been my former boss John Wookey who said “Software development is a lot like making sausage. The less known about the process, the better.” How true! And I know John is committed to improving practices at Oracle. But let’s face it: software development at big ERP companies is inefficient. And looking at Microsoft’s woes with Vista it’s obviously not just big ERP firms that have trouble.

So how do these firms build software? Surprisingly, practices haven’t really changed for decades. The state of the art is still classic waterfall-based milestones where thousands of resources are supposed to hit “Design Complete” or “Code Complete” all on the same day. I’ll risk getting flame mail & go ahead and state the obvious: it’s stupid. Delays are built into the model. In my experience human error, communication mistakes, and scope increases were the 3 most frequent causes of schedule delays, and they certainly weren’t the result of incompetent resources. Instead, they were the result of a system so inefficient it seems ripe for revolution. Southwest Airlines, efficiency zealots that they are, would go berserk if they saw it!

I still recall the original target for Oracle’s 10th installment of 11i was very early in 2004, perhaps January. January turned to March, March turned to June, and eventually 11i10 came out in November of 2004. Customers affectionately renamed the release from 11i10 to “11iWHEN?!”

In Procurement, 11i10 was a very big release. I had introduced 2 new products – Procurement Contracts and Services Procurement. We had interested customers demo’ing product in January. But by November I had lost most of their attention, as well as that of Oracle’s direct sales force.

Of course, post-11i10 launch Oracle acquired Peoplesoft & there were a lot of management changes. Heck, now Oracle has also acquired Siebel. But through all this change 1 thing has remained the same: the delays. Now R12 of Oracle e-Business Suite (on which development began in 2004!) is even further off schedule than 11i10 ever was. I hear the latest target for the release is November. Sound familiar?

Just to make my point, consider this: I was still at Oracle, and still heading up Procurement applications development when R12 scope was determined. I selected features from an impressive candidates list and did my best to maximize Oracle’s chances of growing share in the market. I oversaw functional design efforts and early advanced prototyping. And yet even though I left the group in January of 2005 (to head up CRM for a “New York minute” prior to the Siebel acquisition), my organization’s ideas won’t be tested in the market until 3 years after they were envisioned. And by the way, the organization is led by just as much talent and capability as it ever was.

Why, I could probably start-up a new enterprise applications business and deliver v1 product before Oracle even released my prior work. Sort of a mind-bending thought, I know, but true.

By the way, you could always predict the projects that were going to cause a schedule delay. They were either painted with a big red “I’m going to be so-o-o late” sign (e.g. let’s rewrite the Account Generator) -or- they were cross-product and cross-organization efforts. Whenever teams with differing priorities had to come together to get work done there was the possibility of miscommunication, differences in points of view on prioritization, and, to sum it up, risk.

Okay, now I’ll go ahead and say what you’re expecting: THERE IS A BETTER WAY. A former direct report of mine and all-around smart guy Seth Stafford once came up with an idea called “the chute.” Here’s how he envisioned it: you’d have your copious amounts of resources all working on their important efforts. And at any given time you’d select 1 key project or initiative to be in “the chute”. The attention of the entire company would be drawn to the project at that time. Of course it would have to pass rigorous inspection criteria to qualify for “the chute”. Any issues encountered while adding the project to the existing body of work (in this case the eBusiness Suite) would be Priority 1 for the entire group. The project would have a very short, fixed time period to get through the chute and to release.

What’s great about Seth’s idea is it is a fundamentally different approach than today’s “we need to test all this stuff together” mentality. Combining thousands of destabilizing elements into a new build & then trying to isolate problems and iron out difficulties is challenging work. But Seth’s idea poses perhaps a bigger challenge to traditional enterprise applications vendors – it requires a brand new approach to how many layers of stratified and often intransigent organizations operate.

I’m in favor of software development practices that empower individual groups to move fast and yet still ensure the high intrinsic quality customers deserve. I believe there’s an opportunity to foster chaos-based development while upholding processes meant to ensure standardization and high quality.

As always, I encourage your thoughts. If you’re a Procurement manager, do you really care about innovation in software development, or are slow, lumbering 2-3 year releases with incremental innovation good enough. After all, aren’t policy enforcement challenges and other, more political issues, really your key areas of concern? If you’re in IT, how fast can you reasonably adopt change anyway? Doesn’t a release every 24 months free you up for business-sponsored project work? And is the only way to uptake software faster through SaaS? And finally, if you’re in software development yourself, what’s worked for you?


Written by Dave Stephens

04/24/06 6:08 AM at 6:08 am

Posted in IT, Opinion, Technology

4 Responses

Subscribe to comments with RSS.

  1. Dave, I couldn’t agree with you more! If I only knew your thoughts while I was in Purchasing, I would have tried to rock the boat even more than I did.

    My thoughts from a developer’s POV:

    Waterfall is “old school”. It’s been proven time and again to fail in software development because it’s a rigid process (ie. not suitable for change). Software development, on the other hand, is unpredictable and highly susceptible to change. It’s clear to see how waterfall is inherently flawed.

    One answer is iterative development, which is not a new concept by any means. Iterative processes embrace change, and force early risk mitigation and discovery (typically architecture and/or integration points). You build software in short iterations that reduce complexity and provide a tight feedback loop with stakeholders. Don’t take my word for it; there’s a wealth of knowledge on this very topic of Waterfall vs. Iterative development that is well beyond the scope of a simple blog comment.

    While I was at Oracle, I pushed very hard for a shift toward iterative development with minimal success. I think the biggest obstacle in software development for huge (ERP) companies is change. Not to say that you can’t teach an old dog new tricks, but it’s just difficult to get the 800-pound gorilla to change course. Truely iterative processes not only require the development teams to change, but also the functional/product management teams to change. Traditional waterfall is so ingrained into all processes that any fundamental change would require some major support from people way up in the organization. Although I cannot validate this, my gut tells me that most of the people way up in the org are “old school” in regards to software development.

    Just my two cents.

    Jang Kim

    04/25/06 3:01 PM at 3:01 pm

  2. Jang- great to hear from you! I agree with you, iterative development is noticeably better. But I view the biggest obstacle to change for Oracle, SAP, Microsoft, as being their existing body of work. The flawed logic goes -> 1. wow, we have a lot of code -> 2. if we change something we have no idea what will break -> 3. we need to test all the changes together before release. Once you’re philosophically at 3, you might as well go waterfall b/c you’ve guaranteed massive testing cycles & massive stabilization problems.

    Dave Stephens

    04/25/06 3:15 PM at 3:15 pm

  3. Dave, Thanks for posting this entry.

    I’ll second Jang’s recommendation of iterative development based on my personal experience with it. Some background about myself:
    After several years of working in development in various Oracle Applications teams (Business Intelligence, Sourcing) I left Oracle to work for a small software company in December 2003 – been working there since then.
    We use agile development processes and iterative development. This has worked extremely well for us. We are now working on the third major release of our product and in each one of them we used iterative development. Iterations are usually 4-6 weeks long. We have made minor modifications to how we monitor project progress and track dependencies between modules. This is primarily due to the fact that our team size has grown from when we started. I believe thats its in the spirit of agile development that you start with a process and fine tune it so that it works for your team. Some aspects of iterative development that we have found truly useful:

    o Frequent detection of compilation problems in the code through continuous builds
    o Integration at the end of each iteration means the pain is spread throughout the product development cycle and much easier to handle
    o Developers checkin frequently and fix bugs which break the code or some feature (if you have nightly regression builds)
    o QA gets to start testing early in the cycle – as features get checked off
    o Development reacts quicker and fixes P1 issues which QA finds.

    As a result of this we have been able to plan and execute better leading to a more predictable and manageable development process.

    sudhir rao

    04/26/06 12:56 PM at 12:56 pm

  4. Dave,

    Thanks for the mention of “The Chute” concept.


    I think we all love to hate the Waterfall model, and Iterative or Agile development is something many people have come up with over the years. But the way I intended the Chute is actually somewhat different.

    Yes the Chute model has an iterative character, but where most Agile approaches address themselves to the work of small teams, the Chute is all about how a large organization manages its focus of attention. Where the Waterfall really breaks down in large scale organizations is in its dependency tracking — or lack thereof.

    The Waterfall scatters management attention into a lot of parallel threads of development which then roll up through the organization as ever larger and more meaningless lists of projects. The people at the top wind up in interminable status meetings in which everyone tries to give a clean update and excuse away whatever slips occur by placing blame on some other group they have a dependency on — preferably someone not in the room at the time.

    Putting the developers into small agile teams doing iterative development may do a lot for the “local” quality problem at unit testing scale, say, but does nothing to ensure that “global” dependency tracking is handled promptly.

    The Chute is premised on the idea that what gets measured gets managed and that what most urgently needs measuring is a “forest” of dependencies rather than a lot of individual project “trees”. To find the best path through this forest requires identifying projects which cut across organizational lines and are likely to create lots of integration risk. Rather than leave all the line managers to work on their own pieces in parallel and form up a circular firing squad of blame transfer, there needs to be unitary project management responsibility for each such project. To ensure that managers respect the authority of the cross-team project manager, senior management should rearrange the agenda of their regular status meetings to start with updates from these project managers and back up their requests with immediate orders to the various groups to “make it so”.

    The Chute then is roughly the “pole position” in this new status meeting agenda. The first slot is given to the major project with the highest degree of unit-level readiness and overall impact. The longest sustained senior management focus during each meeting is on the success of that project and the discussion centers on its progress. The meeting then progresses through various less-critical items until — toward the end — there is the usual ‘around the table’ review of exception based tracking of the routine projects.

    The Chute also should be reflected in test methodology along the lines Dave described. Most of the organization should be in development mode much of the time, but in local environments. Only the project in The Chute should normally have access to the main test environment from which fresh production software is created. Code is promoted there by the project in the Chute and is tested by QA from across the organization for regression impact. As issues surface in the Chute (which can also be a name for the central test environment), developers anywhere in the organization are paged (literally or figuratively) to treat those as priority one until fixed.

    I view the Chute model as essentially a model of consciousness. How does a large group of neurons work in parallel to acheive unitary consciousness? The philosopher Daniel Dennett, who has written extensively on cognitive science and is perhaps best known for books like “Consciousness Explained” and “Darwin’s Dangerous Idea” once wrote an elegant paper with the wonderful title: “More like fame than television”. The analogy was this: “television” corresponds to consciousness as an attribute of the “top executive” inside the brain who is busy reading all the crappy reports that roll up to him from subsystems. “Fame” by contrast corresponds to sustained focus of many ordinary neurons on a coherent object. We are conscious not because of a “little man” inside our brains watching a TV monitor, but because the brain organizes itself dynamically around a series of focal points of attention. A series of different things are “famous” among our neurons. And as Andy Warhol might have pointed out, this “fame” isn’t lasting, but flits from one thing to another.

    To make a large development organization Agile involves even more than the already enormous task of making all the little subteams agile individually. It involves choreographing all the innumerable “15 minutes of fame” that each subteam’s work needs to have from the whole organization. The Waterfall is a virtual abandonment of dependency tracking and a decision to hide from the choreography problem. The Chute is a simple organizing principle which gives shape to the forest of dependencies and matches natural human behavior (do the thing the boss is paying attention to, try not to let the blame fall on you, etc.) to the problem of prioritizing a vast number of individual tasks.

    Seth Stafford

    04/28/06 4:51 AM at 4:51 am

Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: