Agile methodologies

Here, I'm gathering my thoughts on the broad topic of agile software development. 

Although the term is fully normalised, my experience is that it's not universally well-implemented and well-communicated. Sometimes, a focus on a given 'agile framework' actually restricts advancement in ways of working, which isn't very agile at all. I think our 'agility' is something we can always work at, with an open mind. The original manifesto and principles gave us a lot of room for manoeuvre and taking advantage of that flexibility is an agile thing to do.

Looking back at the manifesto and principles

I find it interesting to remind myself of what is not in the ancient (2001) scripture, as well as what is. agilemanifesto.org

You can feel that it was a reaction to problematic ways of working that were more common at the time. The fact that it feels a little bit dated in places can perhaps be explained by its effectiveness - it worked in that it initiated industry change. 

It has broadly survived the test of time, which is quite impressive. I still feel that much of its guidance remains useful and relevant, even if it's more a reminder than the game-changer it once was. Perhaps that's because some of our basic instincts (order, planning, categorisation, hierarchies, ...) could and still can lead us down a bad path, when naively applied to software engineering. The discovery of agile software development is like the discovery that the world is round - it looks flat to us initially and the simplicity of a flat earth is appealing, but accepting and understanding the more complex reality is necessary for civilisation to advance. Ineffective approaches to software development, which still crop up occasionally, can be attributed to a series of forgiveable misunderstandings, akin to the flat-earth view.

One of the strenths of the original manifesto and principles is that they are concise. They are light on prescription, so the details are mostly up to us. We have plenty of room to question contemporary processes and practices (which might commonly have an 'agile' label) without undermining the foundations. Indeed, that questioning is very much in the spirit of 'agile'.

The slight downside of the openendedness is that the 'agile' label can be lazily applied to anything - more on that later.

The manifesto

The manifesto, shown above, cleverly conveys a value system that features nuance, encouraging us to think for ourselves. Some take-aways:

• 'Individuals and interactions...': This is our starting point. A project is made of people, so bringing the right people together, in the right way, is our first focus in making a project run well. If we get that wrong, process and tooling won't fix it.

  • This gently hints against an old-fashioned production line approach, which starts with a formal, staged process, consisting of rigid roles and hand-offs, hoping it won't matter who occupies those roles, as long as they stick to a narrow, pre-defined brief. We know now that a one-size-fits-all process doesn't really fit anything or anyone very well in the context of software engineering, which is more fluid by nature. Instead, we start with good people, get them interacting and, if needed, let process be designed around them, not the other way round.

• 'Working software...': This is our goal. This is a reminder to focus immediately on practical outcomes over theoretical aspirations. It's easy to spend a lot of time and money on detailed designs, specs and plans, but writing these does not guarantee the tangible outcome, nor does it reliably test assumtions, resolve uncertainty or reveal flaws. The code, however, is the exact design that the computer will read and act upon, and the writing and running of it is where the real value is. It is the most important documentation we have.

  • This hints at iterative development. We always want a working version to be close at hand, so we grow a software product in increments. This method tells us whether we're on track and how far along the track we are. It stops all the risk building up until a big-bang release that will probably go wrong.

  • Shortening feedback loops is a big 'agile' theme. Working software (ideally in prod) is a very important feedback loop.

• 'Customer collaboration...' and 'Responding to change...' are how we get there. 

  • ...

Unfortunate terminology

The term 'agile' implies a binary and allows us to use it lazily: 'this team is agile, that team is not'. 

However, agility is relative and is often rather complex. 'Agile' should imply a direction of travel, but shouldn't really carry any assumptions about implementation. Unfortunately, the reality of normal usage is often the other way around - implied roles, structures and processes, but not necessarily progress.

To give the term any meaning, we need to qualify it with comparison against something, like the less agile state we wish to avoid or the more agile state that we seek to achieve. We also need to be clear about what agility involves: speed, efficiency, flexibility, adaptability. Sometimes, it's better to just use those terms instead.

Here are some ideas that can help us:

• We can try to use the term 'agility' more, just to avoid the trap of describing things as simply 'agile' or not. 'Agility' is more clearly a continuous variable. It lends itself quite well to comparison. 'Does this process improve or reduce our agility?' rather than 'is this process agile or not?' Misuse is still possible, but maybe a little easier to avoid.

• People understandably get annoyed by 'Agile' being used as a noun. Apart from sounding grammatically clumsy, it gives the impression of being one specific thing we can do, when it's not. So, where we can easily avoid it, we probably should.

• Sometimes, when talking about processes or practices, we can easily avoid the vagueries and assumptions of the 'agile' label by instead using the name of the relevant framework (Scrum, Kanban, Lean, XP, SAFe, etc.).

• When stating our aims, sometimes it's best to leave out the term 'agile' entirely and find other vocabulary that is less likely to be misunderstood: modern, efficient, flexible, collaborative, iterative, adaptive, and so on.

Agile software development aimed to solve a lot of big problems

Newer entrants to the industry might never have seen a proper, pre-agile ('waterfall') project, so might not realise just how bad things could be. This is an important part of our history to tell, because 'agile' methodologies are to a lage extent defined by what they are not, and what they are not is the old way of doing things.

Segregated teams of BAs, developers and testers would sometimes sit in completely different rooms, barely interacting. Months would be spent planning and writing specs, then more months would be spent developing, then more months would be spent testing, only find a mountain of problems that take even more months to fix. Cost were high, timelines were long, delays were normal. Releases were rare and extremely risky. A developer and a project manager might be four steps removed in the chain of command and might rarely find themselves in a meeting room together.

It sounds completely insane now, but it came from an understable risk-aversion and a simple lack of understanding of a very new type of work. People wanted to apply structures and processes that felt familiar from other parts of business and industry. 

Imagine a layman's view of the process of constructing a building: produce detailed designs, bring in the builders to execute them, then send someone round to check for leaky pipes and wonky carpentry, then we're done. This kind of process was incorrectly applied to software, thinking that developers were the builders, but they are not - the code that they write is the detailed design and a huge amount of the implementation detail is ultimately in their hands, regardless of how much pre-planning was done.  The developers are really the architects. The 'build' step is the quick, easy, reliable bit, done by a computer. Lacking the constraints of a physical building project, we can repeat our software build step as often as we like. 

Similarly, concepts like factories and production lines are appealing but just don't work, because every software engineering task is different, often requiring creative problem-solving and experimentation. The realisation that familiar industrial processes did not fit at all, and that there was opportunity to achieve much better outcomes with novel approaches, is the root of the agile movement.

A young industry, changing fast.

I would be hesitant to call anything 'best practice' in software engineering and delivery. There might be 'bad', 'good' and 'better', but the definitions are constantly moving.

This is to be expected, because the industry is relatively young and immature, but also astoundingly fast-moving. It's not like civil engineering, for example, where we have hundreds of years of established practice, unchanging laws of physics and new materials appearing slowly, while old materials rarely go away. In software, we only have a few decades, and each of those decades has been completely different. Plus, the 'materials' of our field are fundamentally different.

Recognising the fact that we are still on a journey, still evolving, feels very important to the topic of agile ways of working. We shouldn't expect any given practice, process or framework to be eternally optimal, but we can see them as stepping stones, hopefully leading us in a positive direction. Sometimes good ideas lose relevance and better ideas emerge. An 'agile' label certainly shouldn't stop us from thinking critically and imaginatively about how we work, or could work.

Is our problem temporary or permanent? Is our solution rigid or flexible?

A process is meant to solve a problem, but we need to remember that some problems are only temporary. We often need to remove or change a process, once it has done it's job.

Even if a process solves a permanent problem, some flexibility may be beneficial and a better process may one day appear.

An analogy: If health and fitness is your problem, you could try to fix it with a strict exercise regime and diet. That's a new set of temporary processes. Once you are fit and healthy, your processes will need to change. Maybe your next goal is to maintain your fitness, rather than transform it, which is probably easier and doesn't need such a strict regime. Or maybe you want to use your new-found fitness to learn a new sport, so you'll start a whole new training regime based around that. Maybe you'll swap Pilates for the football pitch.

We also have stable, long-term processes in our lives, to solve problems that won't go away for the foreseeable future. Hunger will always be a problem and breakfast will always be a good solution. 

But, there is usually more than one viable solution, so we don't necessarily need to be rigid. Brunch, elevenses, and snacks also work. Maybe we can use different combinations of these solutions, as and when appropriate.

Something that frustrates engineering teams is a when a process seems permanently rigid and unquestionable, because there's hardly ever a good reason for that. That lack of flexibility holds us back. Either the problem is temporary and the process should be too, or the problem is long-term, but some flexibility and evolution of the process would be beneficial.

This makes it really worrying when managers have a very rigid view of exactly how something should be done. 'This meeting must always have this frequency, this duration and this scope. This documentation must always have this structure. This team must have these distinct roles.' They might claim to be advocating 'agile' processes, but the lack of flexibility is anything but agile.

Agile frameworks can be temporary solutions, but can become problems

Referring back to the health analogy, I would see full-on Scrum, for example, as being like the intensive exercise regime and diet, designed to solve a problem, but not a desirable long-term way of living. 

(Some would point out that Scrum can be more flexible and nuanced than I imply here, but I'm referring to a pretty common implementation of it.)

Scrum comes from the very early days of agile software development (90s) and is, in my view, a pretty good training course for breaking bad habits and introducing new behaviours. However, I would be very cautious about its potential misuse and I would have serious reservations about using it religiously, long-term.

• Coaching: The 'scrum master' is meant to be an agile coach, whose main job is to guide the team through practices and processes that are completely new to them. They are not a manager or a leader of software delivery, just a coach.

  • But once the scrum processes are familiar (and perhaps not even needed) the scrum master's value reduces.

  • The term 'scrum master' is often misused, when people really mean project manager, delivery manager, team lead, or something similar.

• Iteration: The work planning time boxes (sprints), often ~2 weeks, introduce people to the idea of not planning months or years ahead, but being more focused on iterative delivery and being more adaptable in the longer term.

  • But once iteration and short-term planning become the default, the rigid two-week time box and scope batching can become arbitrary and potentially limiting. There can be additional efficiency in a more flexible approach (e.g. kanban).

• Breaking down big problems: Story points teach teams to assess the uncertainty and complexity of upcoming work, so they are encouraged to break down big problems into manageable tasks and identify risks. One of the aims is to avoid embarking on tasks that turn out to be surprisingly complex, slow and costly, potentially outweighing their value.

  • But once good habits are instilled, the costs and frustrations of a structured estimation process are not necessarily justified. Something more flexible and informal might achieve exactly the same result, with less delay. The estimates are inherently a very rough forecast anyway, so only have a limited value (more on that below). As soon as a task starts progressing, we get real information that supersedes the theoretical estimate anyway, as long as we're communicating.

• Understanding our capacity: In theory, story points and sprint commitments allow a team to focus on a goal, get a feel for what's achieveable in a sprint, so that they and their stakeholders can have realistic expectations, avoiding disappointment. They may also hope to see increases in their capacity over time. These are valid aims...

  • However, estimates are inherently inaccurate, because our work involves problem-solving. Our tasks carry uncertainty that is only resolved in the doing. Any insights we gain from analysing made-up, error-prone numbers are going to be rough at best. If managers try to use story points and sprint commitments as incentivising metrics, they only create perverse incentives that are counter-productive (overestimating, undercommitting). They also distract the team's attention away from what should be motivating them, like product quality and customer value.

  • Furthermore, if we accept the very agile idea that we need to be able to respond to changing information, requirements and priorities, then we need to accept that such changes won't always conveniently coincide with sprint planning day. The reality of our work is often rather different to the rough ideas written on tickets days, weeks or months before. Story point estimates go stale, new tasks enter the board and sprint forecasts go out the window, with good reason. That doesn't mean we're not agile, it means we're more agile than the process ever anticipated.

• Retros

  • ..

• Planning

• Refinement

When people criticise 'agile', or describe it as being like a series of two-week waterfall projects, my impression is that misuse of Scrum is usually to blame.