Testers: what’s your strategy in a continuous delivery context?

In my last stint as a tester from October 2012 to Jan 2014, I helped my organisation at that time moving from delivering once every month, to delivering multiple times a day.

Let me first clarify that we didn’t move to multiple deliveries per day just for the fun of it, but because we needed it.

Your organisation might not yet know it needs this level of agility but more than likely it will at some stage in the future.

How did this transform the role of the testers within the organisation?

Massively

The start

When I joined I found scrum teams that delivered either once a month or once every 2 months. The teams had 3 different defects management databases full with old and new defects. Testers were doing the following activities:

automation (~30-50%)
exploratory testing (~50-70&)

The batches were big, the exploratory sessions were long and found a lot of defects. The automation was not effective, as it was slow and unpredictable, its value was negative.

When I left

When I left, we were using kanban, delivering multiple times a day, defects were more or less a myth of the past, no defect management tool existed. Testers were doing the following activities:

Three amigos BDD sessions with customers and developers
Exploratory testing (1~5%) – never longer than 10 minutes per card, more often than not reporting no defects
Pairing with developers
Coaching developers on testing
Writing automation (0%)
Talking to the customer and the team
Improving the system
Designing the product with the team and the customer
Helping define what to monitor in production
Any other valuable activity the team needed them to do

As you can see the activities that before occupied 100% of testers time, now occupy from 1 to 5% of testers time.

Were testers busy before? Yes, absolutely

Were testers busy after? Yes, absolutely

Were testers complaining because they weren’t doing automation or enough exploratory testing? No, believe me. Most testers I worked with saw the new activities in the role as a learning activity and an opportunity to broaden their skills and become more valuable to any company.

If a tester didn’t want to adapt to the new reality and embrace the change and new ways of doing things, he would have been busy for 10 minutes a day (~2%) and he would have not been useful to the team.

Did we get there with the touch of a magic wand? No, the end stage was the result of many experiments. It was, back then, a good recipe for that context at that time (it is continuously changing)

So, tester, what’s your strategy for working in a company that releases multiple times a day?

What you are measuring is wrong, let me tell you why

September 12, 2016September 12, 2016 Augusto Evangelisti Agile Software development, Lean Software Development, Lean Thinking, See The Whole enterprise, lean, Lean Thinking, problem, See The Whole, suboptimization

There is one measurement that is commonly accepted when examining the effectiveness of delivery teams. Lead Time.

This is a definition from Wikipiedia.

A lead time is the latency between the initiation and execution of a process. For example, the lead time between the placement of an order and delivery of a new car from a manufacturer may be anywhere from 2 weeks to 6 months.

If software delivery team “LightningFast” is able to give to its customers product X in 1 week while software delivery team “SlowAndSteady” requires 4 weeks to deliver the same product we can state without error that LightningFast is better at software delivery than SlowAndSteady. In fact it is 4x better.

(For the purpose of this article let’s consider the quality of the product delivered by the 2 teams to be equal)

At this point we are considering the starting time to be when a development team has some form of requirements to start working on and finishing time when the bits are deployed to a production environment.

As a business owner you would want to hire team LightningFast and try to avoid SlowAndSteady, I agree with you.

Let’s imagine that team “LightningFast” works for “BestProduct.com” and “SlowAndSteady” works for “WeAreTheBest.com”

The CTO at WeAreTheBest.com would willingly spend a lot of money for either replacing SlowAndSteady with LightningFast or coaching SlowAndSteady to lower lead time and become more like LightningFast.

But the business world is not a software delivery speed contest, it is much bigger than that.

Follow me and you’ll discover that even if you have team LightningFast in your organisation you might have bigger problems to solve.

Let’s expand a little our concept and start looking upstream (left).

Team SlowAndSteady have a very effective way of transforming their roadmap into actionable requirements and are able to complete this specific task for product X in 2 days. Team LightningFast don’t have a lot of skills on this department and spend a lot of time upfront to create the requirements. In the specific, for product X it took them 4 weeks.

So let’s calculate lead time one more time.

LigntningFast => 1 week + 4 weeks = 5 weeks

SteadyAndSlow => 4 weeks + 2 days = 4.4 weeks

Okay, does this mean that SteadyAndSlow are more effective than LightningFast? It would seem so. The starting point for the calculation of the lead time changed completely the judgement we had built over the effectiveness of the delivery teams.

This seems interesting, what’s next?

Next is a different question. The question is what is the lead time that we should be measuring? What’s the lead time that is important to our business?

The answer, unsurprisingly, is the whole. From start to finish.

Now, let’s walk backwards to see where the start is.

Next bit I would add is the time that product X took before it got prioritised and given to the delivery team to work on. In many cases, companies have an Enterprise Portfolio from which the Products get selected from when prioritising.

WeAreTheBest.com (the home of SlowAndSteady) have mastered a very good prioritisation process and priorities are continuously assessed based on market conditions, customer feedback and active monitoring so when a product becomes the highest priority for the company the process signals it clearly and they can start creating a roadmap within 1 week.

BestProduct.com (the home of LightningFast) are struggling with prioritisation, they are not aware of how their customers use their products and have no intention of using customer insights to make decisions on what to build. They rely on the CEO that is the smartest person in the company to decide what gets prioritised. The CEO got it wrong this time and should have started working on product X last year when it was added to the enterprise portfolio. It took 1 year for the product to get picked up.

Ok. Now things are interesting.

BestProduct.com => 1 week + 4 weeks + 1 year = 1 year and 5 weeks

WeAreTheBest.com => 4 weeks + 2 days + 1 week = 5.4 weeks

Wow, more than a year difference between the 2 companies, this is incredible!

We could go on and on going back to the moment the idea appeared in that company for the first time and add the time lag between the idea appearing and the corresponding product materialising on the Enterprise Portfolio or even go back further and research when for the first time social conditions emerged a problem to be resolved for customers that will be eventually resolved by product X.

I have seen many enterprises and many delivery teams. Don’t take my word for it, look out there, delivery teams effectiveness is equated to how good they are at doing the Requirements to production transition.

This creates a need for improvement localised to that context. A lot of money is invested and spent to resolve the last bit of the ride, while monstrous inefficiencies slow down delivery in measures orders of magnitude bigger.

the narrow focus on a sub-system is called sub-optimisation and it is a very well known concept in systems thinking and lean but it seems to have flown over the heads of most of the agile community.

Still today, a lot of agile experts focus almost exclusively on technical aspects and are not concerned with resolving the real problems in the system.

Do you want to know more about this?

Do you want to know how to identify the real problems in your system?

Give me a shout and let’s have a chat!

3 Simple Steps to Help Overloaded Teams

September 6, 2016September 13, 2016 Augusto Evangelisti Agile Software development, Agile Team, Culture, Kanban, Lean Software Development, Lean Thinking, War Stories agile, change, coaching, kanban, lean, learning

dublin-rain-491-390x285 It was a wet and dark morning in Dublin when I sat with team X for the first time.

I had been told: “Gus, these guys are struggling, they really need your help. They are always late, and everybody complains about the quality of their work, it’s bad, very bad”.

I expected a demotivated team of technically poor developers with its members playing video games or youtube videos instead of doing their job, but I was wrong.

No developer was slacking or pretending to work, on the other hand, I saw people with worried faces shuffling around, some listening to angry customers, some head down into code, some testing and quietly cursing.

To a newcomer, they looked super busy and doing their thing.

I perceived the first hint of trouble later in the day. I went to talk to some of them, just introducing myself and telling them that I was going to work with them to try to make their life easier. Each and every one of them was too busy to talk. All of them had something urgent, being a production issue, being a customer waiting behind their backs, or a release they had to finish by 5pm. Sorry, sorry Gus, I really can’t now, I have this blah blah…

I took it in and let them alone for the day. The day after, I kept on observing without saying a word. The amount of pressure that I saw applied on to these poor kids was frightening. In the same day I saw 8 different people go to the same developer and demand he finished something he was meant to have finished already. The requesters were different people and asked for 8 different things, all quite angrily.

The subsequent days, my observations stressed more and more this situation, the people were pushed over the limits and by trying to finish things quickly and relief some of the pressure were skipping steps and introducing new problems, new issues to work on, and so on.

I kept on observing and asking questions here and there, but the people saw me as somebody that couldn’t help them with the code, hence quite useless. I had to do something different.

On the 4th day, in the morning, after their standup, I told them to stop doing what they were doing and come and have a chat with me.

Believe it or not, It was almost impossible to get them off their seats.

no-thanks-were-too-busy They felt they could not leave the sinking ship, they needed to continue trying to flush the water out with their hands. Did I have a powerful drain pump and could save the boat easily? They didn’t have time to learn how to use it, they needed to keep on shovelling with their bare hands.

I eventually got them into a room. We had a retrospective. I disguised the retro to be something else, because I had heard from one of the developers: “we don’t do retrospectives anymore, what’s the point if we don’t have the time to change anything?”

That’s a very valid question. Very.

The team was obviously overloaded and didn’t have the maturity and the necessary negotiation skills to express that to their customers and stakeholders. They had reacted to the overload with what I call the “headless chicken approach”. It works like this: “Just do all you can, forget about quality, process, product, customers, just run for your life and even if your work product is terrible, people won’t be able to blame you, as you work like a donkey every day.”

fuck-it-let-s-just-panic-and-run-around-like-headless-chickens Do you think this is uncommon? Well think again.

The first step we took was to visualise the work in progress. The guys were using Jira with no physical board and didn’t realise how much work they were really taking on. The board was scary, full scary I mean.

As the work came from many different products, another thing that we did was separating the products into different classes of service.

By doing this we immediately saw that one product had most of the tickets, why? We don’t know yet, but at least we are aware of it now.
After this very simple step the team members started having conversations that were beyond the need to finish MyNewFeature, they started looking at current priorities and talking about the whys of the bad situation they where in.

Then they started talking about experiments to fix it.

They were improving the system.

I looked outside the window, the sun had come out.

p9220189_dublin_hapenny
What were the 3 steps we made that enabled the behavioural change?

Acknowledge we were in an unsustainable situation that needed changing
Agree the team had the responsibility and the authority to improve it with my support and management agreement
Visualise the work they were already doing

Just this small change had transformed a group of intelligent people acting like headless chickens into people that were trying to improve a complex system, not bad for a weeks work I’d say!

If you are curious stay tuned and I will tell you what happened next, no more chickens, I swear.

The new episode of the story of Team X is out

The art of doing 1/15th of the work and getting earlier business outcomes

August 10, 2016August 11, 2016 Augusto Evangelisti Agile Software development, Lean Software Development, Lean Thinking agile, lean, lean startup, SCRUM, waste

If you are an agile practitioner you are likely to have read the book “Scrum – The art of doing Twice the Work in Half the Time” from Jeff Sutherland. I am a fan of Jeff and I believe that what he has done for software development in the last 20 years is great.

I do have issues with the book’s title though.

That title is what makes people think that being agile means being able to go fast and deliver more stuff.

Is this what real agility is? Let me tell you a story.

At a client of mine a couple of years ago, I was asked to coach a product team and help them with a new product they were starting to work on. I was excited about it and asked the product owner to meet for a coffee and initial chat.

He kindly agreed and told me: “before we meet, have a look at our requirements document so you will know what the topic is”. He also told me that a high level estimation had been done and that the product would take from 6 to 8 months based on one agile team and that there would be licensing costs associated with an automated scanning system we had to acquire.

Attached to the mail there was a document of about 50 pages with detailed workflows, low fidelity prototypes of the screens and quite a lot of explaining text. I skimmed through it looking for a description of the problem that the product was meant to resolve, but couldn’t find it.

I read and reread the document and I couldn’t find the original problem that the product had to resolve, it was not there.

When I met the product owner, after agreeing the weather was miserable (that’s how we start any conversation in Dublin regardless of the season) he started describing his solution. I let him explain to me the beautiful features to build and the amazing technologies we were going to use.

When he was finished, I asked him: Why are we building this?

The initial reaction (completely normal) was a defensive stand for his solution. When I probed more, it was clear that after having worked for so long on the product on his own he had forgotten the nature of the initial problem that triggered the decision to build this product.

Using the 5 WHYs technique, in about 10 minutes, we eventually got to the initial problem that we needed to resolve.

At this point the conversation became different.

I asked what he thought were the features we should prioritise to resolve the problem so that the customer could have something earlier than in 6-8 months. That triggered the interest in the PO that identified 5 features (about 30% of the total described in the document).

I then took each feature and asked him if it was necessary to resolve the problem we just identified. After another 10 minutes we agreed that 1 single feature, that accounted for about 1/15th of the initial solution, would resolve the customers problem. To avoid making the PO feel bad about having done so much unnecessary work, I told him that we would build the other features incrementally, and that it was a great success that he had identified a single feature that would be useful for the customer almost immediately.

We then agreed to identify outcomes of success for the full product and start measuring immediately as soon as we released the first feature.

These read something like:

“We will know we have succeeded when 30% of customers do X instead of Y” or

“We will know we have succeeded when we will have 40% less support calls related to problem X”

et cetera. I made sure the business outcomes were related to the initial problem, not any product.

What ended up happening is that we built the first feature in 2 weeks (opposed to 6-8 months) and the measurements told us that we had reached what we wanted already.

We never built the other 14 features, we stopped because the business outcomes we had set were reached.

And yes, you guessed, we didn’t have to buy the licenses for the automatic scanning system either.

As a coach, my mission in organisations is to guide teams and navigate problems, maximising value with minimum work. This is always welcome when meeting CTOs or CIOs because – most of the time – less work means lower costs and earlier delivery.

We did 1/15th of the work and got business outcomes earlier, a big improvement IMHO from “doing Twice the Work in Half the Time”, what do you think?

This is how I measure the quality of my work

August 5, 2016August 9, 2016 Augusto Evangelisti Agile Software development, Lean Software Development, Random Thoughts, Uncategorized

I have blogged in the past about product development and measures of success, it is a topic that interests me a lot.

I am an individual that sells services to organisations that develop products. This makes me a product as well. In fact organisations might want to hire me because they have a problem and believe that I (the product) can resolve it.

The product I advertise is something like this:

“I will help your organisation deliver products that matter through coaching your people on the use of agile and lean practices”

There are many ways of defining success KPIs for engagements of this type. targets will be set, numbers will be pulled out of thin air, promises will be made, but I am not 100% sure that the quality of my work can be measured like this.

I might meet or exceed all the KPIs we agreed at the start of my engagement, but if at the end of my gig you still need me to fix the same problem, then I have failed.

A coach must be able to work himself out of his own job.

If you don’t need me anymore because your people are now able to resolve the problems without my help, then I can safely say I succeeded.

So, my promise to my customers is:

I will do all I can so that you don’t have to pay my services in the future

Why are we sprinting?

May 22, 2016 Augusto Evangelisti Kanban, Lean Software Development, Uncategorized kanban, lean, SCRUM

A few weeks back I asked this question on a very popular Agile and Lean LinkedIn Group:

10 years ago not many companies delivered value every month and Scrum really helped the industry with the concept of sprint. People started thinking in terms of vertical slices of value rather than systems and started to deliver value often, it was a great step forward towards agility.

Today most shops do 2 weeks sprints and it is commonly accepted that smaller batches are better than larger ones as they are less risky and also allow for earlier benefit realisation.

I understand how the concept of sprint has helped the industry become more agile and lean, but, and there is a but.

Why have sprints today? Why create artificial deadlines? Why create a minimum batch size, be it one week, two weeks or whatever it is?

If I am a mature organisation, I probably have cut delivery transaction costs and have a good strategy for continuous delivery and I am able to release whenever I want with little effort and risk. Why can’t I release when I have value to deliver?

Can anybody give me a valid reason for having sprints?

The topic became popular with a total of 35 thumbs up and 85 comments (including my replies)

Reading the answers, I am not only more and more convinced that sprints are unnecessary and dangerous artificial deadlines, but I am starting to think that in some agile practitioners thinking, the scrum guide has replaced the true reasons for agility.

If you have better reasons why we should always sprint, please add them as a comment, I am still hopeful I have missed something and want to learn.

Testers Prevent Problems, every day

May 5, 2016 Augusto Evangelisti Lean Software Development, Testing prevent bugs, prevent defects, prevent errors, prevention, Testing

Earlier Today, I read an article from the notorious tester Michael Bolton titled “Testers don’t prevent problems“. I would like to use an example to counter this assertion end expand the conversation.

Disclosure – I am a firm believer in prevention over detection in product/software delivery.

The Fact– In today’s news, we read that the ECB will be stopping the print of the 500 euro banknote because of its association with money laundering and terror financing.

(Source: The Guardian)

The Problem: According to the article, producing 500 euro bills was an error. It is a bug in the product “Euro currency bills and coins denomination”. In fact such bug has caused quite a lot of problems to our society according to what reported in the article.

Larry the tester – Let’s imagine that Larry, a tester, happened to be in the room where people where deciding the denominations of the euro notes. Let’s imagine he said “Hang on lads, this could be a problem as it would be easy to conceal large sums of money and smuggle them. Banknotes this valuable might help illicit traffic, should we stop at 200 or maybe even at 100?”

Let’s assume there were smart people in the room that decided to take the objection into consideration, made research to prove its validity and as a consequence decided not to print the 500 euro notes.

The Question – Could we say that Larry had helped prevent the problem?

I say yes, how about you?

What I like about Jeff Gothelf’s lean UX approach

March 22, 2016 Augusto Evangelisti Lean Software Development, lean startup agile, lean, lean startup, lean ux

Product development has been my focus (more like obsession) for the last few years. I have read all the books and experimented with many of the approaches from lean canvas to impact mapping to the 100 different ways of framing an MVP.

While experimenting I had some success, many failures but in general I have been left with an after taste of something missing. Sometimes I failed to engage by using the wrong language, some other times I failed to even get started because some people just don’t want to hear that they might be wrong and are threatened by experiments that might just show that.

I had the fortune to attend the workshop “Lean UX in the enterprise” with Jeff Gothelf in Stockholm yesterday and I think I might have found a simple tool that can help me frame the conversations and get results.

Jeff is an excellent facilitator. During the workshop he gets teams of people to design a product. I was really impressed by the simplicity of the approach and the universality of the language used. But most importantly, at the end of the exercise, making decisions on what to build first becomes trivial.

His approach is linear, simple, easy to reproduce and will resonate with anybody with some understanding of lean and agile. I have to say this is a breath of fresh air in this space where some of the authors sometimes feel they are not great if they don’t introduce some new unnecessary term that creates ambiguity and confusion (but also a lot of discussion and website hits for them).

I am going to experiment with Jeff’s approach for the next product I see, I really look forward to it. Thanks Jeff! gothelf

What I learned about helping teams use WIP limits

March 15, 2016 Augusto Evangelisti Kanban, Lean Software Development coaching, kanban, wip, WIP limit

The Work In Progress limit (WIP limit from now on) is one of the most powerful but counter-intuitive concepts I had the fortune to encounter in my work life.

When suggesting teams to introduce a WIP limit to their boards, I hear objections like “are you saying that doing less things at the same time is going to make you faster? You must be wrong? If I start 10 things I’ll finish earlier than if I can only do 2 at the same time!”

or “Working on one thing at the time only? That’s NOT efficient!”

It is indeed a counter intuitive concept, proven by the fact that most people object to it based on common sense.

You might say, why don’t you explain the reasons behind the value of WIP?

Don’t get me wrong, I am familiar with queuing theory, Little’s law and the relation between resource utilization and queue size, but these concepts are not something that can be explained easily every time that the objections are brought forward. Funnily enough, explaining them, is not really efficient.

I have been helping teams use Kanban and the Kanban method for a while now and I have used different approaches around WIP limits to identify the one that gives the best results for the teams.

I have come to the conclusion that the most efficient way to go about this is by (I hate the word) imposing the WIP limit at the very beginning of the adoption, as part of Shu in the Shu Ha Ri learning pattern.

The great thing about having a WIP limit is that it gives strong signals to the team

What’s the most important problem we have to solve now?
What’s the highest risk our kanban board is telling us about?

If your board has a WIP limit you will have to answer the questions and act, if you don’t have one, you can always pick the easy cop out of taking a new less important task
The questions posed by the WIP limit are uncomfortable ones, they won’t allow the team to take the easy option. That’s why I believe that teams will benefit from having them since day one.

I have seen WIP limits cause turmoil in teams, people initially feel uneasy. I believe, the feeling derives from the fact that team members don’t believe yet WIP limits will work, and this is due to their aforementioned counter-intuitive nature.

Eventually teams will internalise the fact that WIP limits help the team focus on delivering instead of retreating to more comfortable, less valuable tasks and it does indeed improve flow. But that’s after a while.

My point is, that if we don’t impose the limit at the beginning, the team might not get there on their own.

As I said before I tried many different coaching approaches, the one that worked for me better so far is this

a) A session with the team in which I go into the details of the mathematical reasons why WIP limits improve flow (most people won’t believe everything but will have some context and hopefully some curiosity)

b) Get the team started with a system that includes a WIP limit initially set by me and low enough to be uncomfortable (see above)

c) Attend team’s standups, to discourage breaking WIP and ask the difficult questions I mention above if the team is not able yet to read them from the board

d) Facilitate a retrospective after 2 weeks of use and have a conversation around whether the limit is too low or too high

What was your coaching experience? Can you share alternative paths?

#WhyScope – Looking for better measures of success

February 23, 2016February 24, 2016 Augusto Evangelisti #WhyScope, Lean Software Development #WhyScope, Budget, Cost, lean, Product Delivery, Success

I’m sick of it. Sick of hearing the scope of this, the scope of that, it’s in scope, it’s out of scope, scope creep, full scope, MVP scope, we need more developers to get the full scope in time, we need more time to get the full scope with these resources. Scope ad infinitum.

Why are you so upset, you might ask?

The reason is simple: thinking in terms of Scope, Time and Budget encourages bad behaviours.

For example, delivering a pot of crap filled to the brim within timeline and budget is considered a success! Yu’ve got to focus on timelines and accurate estimates, you are not required to give a shit about what the customers really need but just to make sure you deliver the full scope, to the brim.

(Image from Claudio Perrone @agilesensei stolen from this beautiful piece of work that everybody should see)

Instead, if you think out of the box, happen to talk to the customers and understand that what you are building is actually a pot of crap that they don’t want, you have failed, your product didn’t deliver the full scope, you are a failure, shame on you!

I don’t want to work in an industry that thinks in these terms. Am I quitting? No? I want to change it.

Do you want to help me? Start thinking about ideas to shift the conversation from scope, time and budget to measures of success that are meaningful. How about value to customers? How about ability to adapt to their demands? How about development team’s happiness? How about <%add your idea here%>?

Tag these ideas #WhyScope and let’s discover a better way to describe success for a product delivery team.