Since Ada Lovelace wrote the first (depending who you ask) piece of code for a computer, software development has been an enigma. A strange mix of science and logic yet a touch of magic in seeing the result of the machine crunching numbers.
Software development is clearly a technical process with its specialised languages and syntax, digital tools and a whole suite of nerdy sounding topics. Along with that deeply technical aspect there is something very creative about building software, there’s an art and coders often have recognisable coding styles (much to other coders dismay). It’s perhaps this juxtaposition of ideas that has made software development a hard thing to pin down as a discipline. Over the years there have been many attempts at controlling the seemingly chaotic process in delivering software. We often see attempts to distill the process into something that can be given an acronym or a management process that can predict what software will look like and how long it will take to create. We even see attempts to remove the coder as if the human in the system is the core problem.
The traditional “Waterfall” approach to software development treats the process as a strictly engineering problem. Someone tries to create pages and pages of technical specifications with all the details needed to set in stone what the development team should build. This specification is “signed off” as contract between a software development team and the customer. The coders go off and feverishly work to deliver the specification as closely as possible in the agreed time frame then reveal all after they have reached the end. Often only then to find the customer no longer needs what they asked for or they changed their mind or the requirements were misunderstood in the first place.
There are several gaps with this Waterfall approach:
- How do we know we captured the requirements correctly?
- How do we know that the customer even knows what they need to solve their problems?
- If we have misunderstood the requirements when do we find out?
- What if the business drivers for the project change and the specification needs to be changed (even if it was perfect in the first place)?
- What if the requirements are correct but the development team misunderstand them?
This is an extreme illustration of Waterfall and some proponents argue that regular demos and customer input can head off issues before they get out of hand. To a certain extent that is true but then why waste so much effort creating a detailed specification at the outset to only throw some of it away in the middle of a project? Although rare; in some niche products Waterfall will work – highly restricted domains where failure is catastrophic in nature and puts human lives at risk for example.
The core reason Waterfall development tends to generally fail is that software is made for human use. At some level in an application a human is trying to get some value out of the system. Humans are messy; they tend to forget things or change their minds. They tend to learn new things and have new insights as they gather that knowledge. People tend to disagree with each other and getting alignment between a group is hard. Just the process of working through a Design workshop can help a customer understand their own domain more deeply and change their ideas about what they require in a solution.
By this stage if you have had any experience in Software Development at all then you have heard all about Agile. In short Agile Software Development encompasses a set of ideas defined by a group of 12 software experts.
The core values of the Agile movement are:
Now this might seem a little esoteric but in essence it nails the idea that software is made for humans and it’s a collaborative creation between those that want the software and those making it. When we create software it’s a process of collaboration with other people; coding their knowledge into computer systems that help them get meaning and value out of data. There are interesting analogs with Design Thinking and it’s ethos of empathy, user experience and the journeys they take (a topic for another blog post!).
The process of Agile varies between different flavours and ultimately the theory encourages teams to take the core tenets and adapt them to fit the project challenges they face. There is however a broad consensus that tight iteration of development effort and delivery to the customer for acceptance testing along with “just in time” mentality defines this approach.
Rather than create a hefty specification that is out of date as soon as the file is saved to disk Agile describes broad activities the user will perform as “Epics”. These Epics are from the user perspective and written in Domain Specific Language to match the user thinking. These Epics are further broken into smaller scale User Stories and then to smaller tasks if required.
The other main difference between Waterfall and Agile is in the project delivery. Waterfall tends to have long running processes to deliver swathes of the specification whereas Agile breaks down the effort into a series of time boxed scopes. This time boxing is called a “Sprint”, each Sprint is planned, estimated, delivered, tested and ultimately handed to the customer. The real power of Sprints is that they are typically counted in weeks (most often 2 weeks in duration) as compared to the months that might pass between demos in a Waterfall approach.
The pros of Agile are:
- It encourages collaboration between customer and development team by making the customer part of the team itself
- Tight feedback loop between development teams and customers
- The development team can only “get it wrong” for a maximum of one Sprint until the next delivery of the software
- The priorities of a project can be changed after any Sprint (whether that’s new ideas or business drivers)
- Documentation and planning is done “Just in Time” so a change of direction doesn’t waste too much specification effort (there are direct parallels to Lean and Toyota but again a topic for another blog post)
Agile has been around long enough that it’s made it into the vocabulary of most software organisations and all the way up to “C level” in many places. It’s one of those topics that when someone starts to talk about it people tend to roll their eyes and take it for granted they know all about it. Interestingly Agile is actually very hard to do right and most companies tend to get it wrong. Companies tend to end up using some form of Waterfall approach that is delivered in an iterative way so it appears to meet the Agile definition.
Things that can indicate when your Agile isn’t Agile:
- The customer is never involved directly with the Development team
- The specifications are written in the form of a long list of demands
- The specification is never re-visited
- Demos are never done
- Work is done in a “silo” where the specification is delivered to the coders and the code is delivered to the testers without and collaboration
- Estimates and delivery promises are made without the development teams involvement
The list above is not an exhaustive list but does point out some of the tell tale signs of “Iterative Waterfall” process. The reason many organisations fall into this trap is that Agile is actually hard, really hard. Having razor focus on the customer and allow them the space to change their mind is hard. To resist the temptation to make everyone agree on what to do and not change it is hard. To believe that a development team can be trusted to speak directly to customers is hard. This difficulty has to be faced for the same reason Waterfall tends to fail – software is made by humans for humans. It’s an intensely creative and collaborative effort to get it right and that is hard.
There are several cons of Agile (depending your mindset and job role):
- Agile makes it very hard to estimate the total effort required for a project (and have an answer for upper management)
- Changing priorities can fluster and anger the teams involved (especially if code gets thrown away)
- The management overhead in Agile is much greater due to the intense stakeholder management required
- It requires serious buy in from customers as the demand on their time is higher than waterfall
Ultimately though there are ways and approaches to minimise the drawbacks while gaining all the benefits of Agile. We can make rough estimations of effort and prioritise the work according to the highest value. By working in the customers priority (and allowing them to change priority for each sprint) we tend to find that parts that are left if we run out of budget are low value so could be dropped. If they are high value then it’s easier to get an extension as the work that was delivered is even higher value and driven by the customer,
This level of openness and collaboration between customer and development team removes the shock of over running and makes “end of project” a much happier time all round.
Is it worth it?
The truth is that building software is an inherently empathic thing to do (as per Design Thinking). Software is built for humans and as such the process has to take into account the intricacies and oddities that come with humans. We can’t specify everything in every detail at the outset because the people using the system can’t fully articulate what they want or need. We can’t say exactly how long it’s going to take because software is never really “done” it’s only every good enough for people to get value from. With all that said Agile has proved time and time again to deliver more happy customers than Waterfall. Each methodology has its place and strengths but more often than not I recommend the first port of call should be Agile and a customer centric approach.
Agile does require buy-in, trust and some bravery from management in believing their teams can get it right (and talk to the customer appropriately!) which can often be the biggest hurdle for success and adoption.
Astar Digital can help introduce, coach and trouble shoot agile. Drop us a line using the Contact form on our site and let’s talk.