What percent of your software development projects come in under budget and on time?
Sounds like the setup to a joke, right? When leadership hears estimates for software projects they automatically double them in their minds. Then they listen to methodologies and promises, praise the optimism, and then change requirements five (fifty?) times during the project. Then 31.1% of projects are cancelled before completion and 52.7% of the remaining work costs 189% of its originally projected cost (The 2014 CHAOS Report from The Standish Group).
If it guaranteed delivery, many of us would accept a cost 189% of the original projection immediately.
Where there is major waste and major pain, there is opportunity. I propose a new curriculum line for code academies like Galvanize and General Assembly focused on educating non-technical leadership on the basic realities of software development.
Let’s take a step back, though, and try to lay some common ground over the root causes of the colossal waste documented by the Chaos Report and many like it.
I am new to my neighborhood in Austin.
We bought a newly constructed home; when you look at my address on Google Maps you see an empty lot, with a Porta-Potty and a hand-drawn sign on a tree reading, “DO NOT DAMAGE THIS TREE.”
Until recently a neat, older ranch house with a wild succulent garden sat a few doors down, next to a stream that runs through Dawson and down to Lady Bird Lake. The owner was already gone, we learned. A mosaic artist, she had used the bridge over the stream as a canvas.
The house was soon bulldozed and the giant prickly pears, the agave, the burro’s tails with it. The bridge remains, sparkling and chipped.
As is often the case in South Austin, the land with a single home became a condo, with two structures: ‘A’ and ‘B’. Both single-family homes. Watching them go up reminded me of the advice a superb software developer gave me when I was leading my first large-scale software project (vs. hacking stuff together in the metaphorical garage). We weren’t making much progress, and since the project was a complete re-architecture of a system used by many schools, all of whom needed and wanted much more than the existing solution provided, the goal wasn’t just launching an MVP and learning from the market. We had to deliver a polished solution.
He told me not to worry, that software was like building a new house: it didn’t look like much was happening to the outsider, but once the frame went up immense, visible progress would be made.
He was right, provided careful thinking has gone into the design. Watching the progress on my street, from concrete pad to framed houses with siding and a roof in a matter of days, has dramatized my friend’s analogy.
I walked my dogs past the construction site yesterday. There was a guy at ground level with a table saw and a guy on the first floor roof with a nail gun. Table-saw guy cut parallelograms to fit, then tossed them to the guy on the roof. Roof guy caught them like you’d catch a frisbee, sandwiching them between his hands. Thunk thunk, buzz buzz. Visible progress by lunch. Roof covered in plywood before quitting time. House on the market in 6 weeks.
Software is rarely like this.
Imagine, with these houses framed out and windows in place and materials precisely calculated on site to finish the job, the construction manager came by and said, “we need to add a third floor.”
Or, “we need to move the first floor powder room to the back of the house.”
Or, “we need a two-car garage.”
The neat, workmanlike progress would stop. The guy on the roof would leave his state of flow behind to stand over blueprints, talking to the manager and table-saw guy. Forward progress would stop – why cut more parallelograms when the work may be wasted?
Roof guy would ask good questions: “if we move the powder room to the back, is the new owner okay with the second floor master bath moving too?”
“That’s a good question,” the manager would reply. “I don’t know.”
“Do they still want a bathtub in the first floor powder room, or is it going to be more traditional?”
“I’m pretty sure they are okay going to a smaller footprint with just a commode and washbasin,” the manager says.
“We could do a shower in there,” the roof guy suggests.
“Does this customer want their house by August, or are they okay waiting until Christmas,” table-saw guy asks (There’s always a fantastically crusty, pragmatic, love-em-while-you-hate-em engineer/builder on a team).
“They need to be in by the start of the school year,” the manager replies.”Good luck with that,” table-saw says.
The manager looks nervous but waits. He needs the changes and a promise of August move-in. He needs to relieve the short-term pressure put on him by the buyers and his management.
The guys who do the work look at each other; they’ve been in this spot a hundred times. It’s the situation that makes them long to be lawyers or plumbers or crime scene clean up guys. Or to run their own construction company. Anything but this ignorant, ethically dubious, sign-your-name-in-blood scenario.
If the project was their project, if they were owners and set to reap the full benefit of their hard work and promises kept, then maybe. They could work into the night, weekends (they probably already are, but they could work 14 hour days). They could get it done, maybe.
But they’re paid by the hour. There is no IPO coming on the two houses by the stream in South Austin. Mark Zuckerberg can’t lock them in and expect a quality output. They don’t know what problems they might encounter, promising something on the fly that hasn’t been deeply thought out. They don’t know what other requests will come; agreeing to a massive change so late in the game is like feeding a dog table scraps. A dog figures out the rules pretty quickly and soon meals are a wreck of whinging for more, more, more.
Of course this scene doesn’t play out in construction, at least in Austin, Texas where city permitting processes are expensive and time consuming. Building a house in Austin is a waterfall thing, not an agile thing. Which is a form of protection, for both builder and buyer. Specifications must be defined way up front and then the workers can get busy.
Most likely the buyer wanted, more than anything, to be in their house before the start of school. If they really really wanted a different sort of powder room they should pass on this house and maybe rent for a year. Or be at peace with the powder room they get.
But software, particularly to the non-technical, has no such safeguards. “How hard would it be to add a button to do XYZ”, leadership asks. On the surface, not so difficult. But what about the overall UX implications? Does “adding a button” on one screen change the metaphors and patterns used throughout the application? Does that button touch 5 different areas of the application? Areas that were not designed with that button in mind? Areas that will all have to be updated, then tested for unintended consequence? Are those areas more brittle, perhaps older modules that haven’t seen much love recently? Does the current engineering team even have familiarity with those sections of code? Will they change something without knowing they’ve changed it? Resulting in a Sev1 or Sev2 bug that will scramble the entire team for three days of frantic work to find an edge case causing widespread unhappiness?
A house in South Austin is a predictable thing. You can look at the house plans. The plans are 98% the same as the other 50 houses you’ve built in the last two years. No one is going to decide to switch from using nail guns and siding to some new adhesive the architect heard about at a conference. Or substitute asphalt shingles for Elon Musk solar panel roof tiles.
So how to reconcile the world of executive/business needs with the reality of software development? And how to leverage the genius of software: unlike a house, if designed well (in modules with clear inputs and outputs) software is easily extensible. So the little bungalow on 0.1 acres you build before kids? In software, that bungalow can grow to 3 acres of manicured gardens and a 4,000 sf MacMansion if you’d like. Or a 200,000 sf warehouse with retail and a helipad.
When non-technical leaders (aka ‘suits’) are reconciling the needs of the business (burn rate, promises to customers, requirements for differentiation, promises to prospects, technical debt) with the look in their engineering team’s eyes, they need a deep understanding of these scenarios. They need to trust and they need to ask questions a few levels deep.
Most of all, I’m convinced, they need some experience they can reference. This is where the business opportunity for the code academies comes in…
Education companies like Galvanize and General Assembly should offer 3-day ‘software development for non-technical executives’ courses. Make them experiential and team-based. The suits should CODE. Even the simplest of projects. Design the project to have ‘discoveries’ that change estimates. Get the teams together at 1pm on the second day and make them estimate how long it will take them to complete the work. At 3pm throw some new requirements into the mix. At 8am the following morning pull them away from coding, again, for another estimating session. Bake in some landmines, like a bug in the payment provider’s APIs that block progress for 90 minutes. Charge $10,000. The ROI for a company will be off the charts (from the same CHAOS report: the average cost of a software project at a US-based medium company is $1.3M).
This isn’t to say software is art, and we have to let the artists take as long as they take. Waterfall vs. agile is a false dichotomy. Waterfall is largely a bogey invented by agile as a target for rock throwing. Outside of large, consultant-driven projects this sort of work has always been agile to some extent or another.
The mosaic artist’s name is Stefanie Distefano. I don’t know her, but I do appreciate her bridge every day. I bet she would understand the confluence of art, invention, and business needs that comes with a software project. She is a working artist, with many clients. If she’s mosaicing a patio I don’t imagine she can tell the owners that she’ll be there for between 3 and 22 days. And probably that first public project, the bridge on my street, had a lot of discovery (read: delays).
But now she’s a pro. She can see around corners, anticipate challenges. She sets the project scope and does the work. She’s product owner and engineer and account manager. Which, like most things, is a blessing and a curse. Bigger projects require specialization. To be successful (on time, on budget, delighting customers) the various roles need to be in balance. The asymmetry in power between development teams and leadership can cause grievous harm to outcomes in software development. Which is just another way of saying that there is a whole lot of education required for these two worlds to work in sync vs. in opposition or in ignorance of one another’s perspectives. I think engineers understand the world of business needs pretty clearly. It’s the suits who need to do some work. The good news? It’s completely teachable. And I don’t think it takes an MS in Information Systems to get there.
El Paso Street Bridge – Austin Texas
