How Hard Do You Want To Land The Plane?

Risk Based Testing or Risk Based Delivery?

Through our 2i Communities of Practice we had a recent session with colleagues to focus on Risk Based Testing. It’s an approach we’re all familiar with but have also all encountered challenges at one time or another in implementing effectively, which we wanted to explore further.

We posed two questions for the group to answer;

1. What is Risk Based Testing?
2. What are the challenges to implementing effective Risk Based Testing?

Through exploring these two questions and sharing our experiences – good and bad - we’ve established key considerations to ensure your organisation is equipped to implement effective Risk Based Testing.

Risk Based Testing is a component of Risk Based Delivery

Risk Based Testing is a widely adopted approach, used to schedule and prioritise test activities considering the importance of the function, technical complexity and impact/likelihood of failure for the change being delivered.

One of the common challenges that we see is that the risk assessment, scheduling and prioritisation is only performed once the project reaches testing. In many cases this is to address an issue that has already occurred, maybe when a plan is off track and there is a need to refocus efforts. This isn’t Risk Based Testing – it’s time boxed testing.

Effective Risk Based Testing is possible only where Risk is assessed and understood throughout the delivery lifecycle. It’s not an activity isolated to testing, but an approach to change delivery from the outset of requirements analysis, through design and build culminating in a suite of tests which are aligned with risk appetite and a consensus on what is important and “good enough”.

Focus on Business Outcomes

 A key component of understanding the impact aspect of any risk is determining the importance of the function to be delivered.

In change delivery terms that requires a clear understanding of the business outcomes being delivered and therefore the business value. Within the outcomes being delivered there will also be some that are of greater importance than others which must also be agreed and understood.

Understanding the business outcomes and value, coupled with prioritisation of those outcomes, means the delivery team know why they are delivering a function and they are better able to work towards a common goal. Focus can more easily be directed to where there is greatest business value throughout the lifecycle and not just once it reaches testing.

Secure Stakeholder buy-In and a common understanding of what is “Good Enough”

A genuine Risk Based approach requires an investment of time and effort from the delivery team and key stakeholders at the outset of the project to identify and assess potential risks. This isn’t a one-off exercise though and should be undertaken throughout the delivery to reassess the risk profile at various stages of the delivery lifecycle.

We’ve seen this being viewed as an unnecessary cost of delivery but can you really afford NOT to do it? Your priority at this stage MUST be securing stakeholder buy-in and gain appreciation that this investment adds value to the overall delivery - increasing certainty of delivery and ensuring the product meets the agreed quality expectations within an acceptable time to market.

Securing the stakeholder buy-in can be a challenge, however. How do you convince stakeholders that time spent before cutting a line of code will improve overall cost, quality and time in the long term?

How hard do you want to land?

A colleague has a great analogy that helps secure buy in and start the conversation on what constitutes “good enough” – He poses the question “How hard do you want to land the plane?” It’s an analogy that we can all relate to and does not need a technical understanding of how change is delivered to see the clear relationship with risk.

Project change delivery is like a pilot bringing a plane in to land. Depending on risk appetite he can bring the plane down smoothly with no casualties – it takes a bit longer, costs a bit more but the quality is high. Or the pilot can bring the plane in to land more steeply, it’s a bumpier ride there are some casualties – this takes less time, costs a bit less and the quality is lower. I’ll make a guess that nobody wants to be on the plane that runs out of fuel and doesn’t even make it to the airport.

So, what are your stakeholders more comfortable with?

Do they want a product where quality is paramount and are satisfied that to deliver the quality expectations that it might take a bit longer and cost a bit more? Or is time to market or cost the driving force and quality is of lesser importance? The plane analogy above is your risk appetite described in life-threatening technicolour!

In reality, many deliveries are somewhere in the middle but it’s a very useful analogy to drive the conversation – What is “good enough” for your organisation or delivery team? What is the culture and risk appetite within your organisation?

Gaining a common understanding of what is “good enough” in terms of cost, quality and time aligned with stakeholder risk appetite enables the whole team to deliver with this in mind and informs decisions as the delivery stage progresses. 

Risk Management throughout delivery

Risk Management cannot be a one and done activity. It’s not a case of identifying risks, writing them up on the register, putting it on a shelf and forgetting about it.

It must be an ongoing activity throughout delivery – considering Business, Technology and Operational Risk. As risks are identified and investigated, information is uncovered, and this informs the delivery and decisions taken.

This can uncover new, previously unknown risks, or increase confidence on known risks or highlight that a known risk is closer to becoming an issue. The delivery process needs to be able to adapt to take account of this.

Risk appetite will also be influenced by the time to correct issues. If time to market for your organisation is key and the organisation is able to release change to Production frequently, the risk appetite may be high - if an issue is found it can quickly be corrected. However, if time to market for your organisation is slower and releasing change to production is infrequent or complex the risk appetite of the organisation will likely be lower as it will take longer to resolve any issues identified.

It’s vital to consider the parameters to be used to measure risks throughout the delivery and ensure risk feedback loops are in place so that informed decisions can be made based on the information uncovered.

The team may need to pivot, if, for example, a new risk is identified when testing a particular function. The decision may be made that a greater depth of testing than was originally planned is required in this area. This decision will have an impact on the existing plan, dependent on risk appetite and what constitutes “good enough”.

If the risk appetite is low, it may be that time or cost must increase to allow further investigation of the newly uncovered risk through testing. If the risk appetite is higher this may mean that a decision must be made to reduce test coverage in another function of lesser importance to protect cost and schedule.

A key part of risk management is also ensuring appropriate mitigations are in place for identified risks. This may be investigation of the risk through testing for those that are deemed important.

For a risk of lesser importance, it may mean mitigations in Production if, (going back to our plane landing analogy) “good enough” means they are willing to accept some casualties.

Identifying and agreeing these mitigants during delivery inevitably reduces the likelihood of surprises when the change “lands” in production.

Drive technical risk down and business confidence up

Ideally, with the risk profile of the overall delivery as a basis, testing can be scheduled to enable time and effort to be focused in the areas of greatest value. Uncovering information to reassess the risk profile also contributes to minimising surprises during delivery and increases your certainty of delivery.

Effective scheduling of tests in this way helps to drive down technical risk, as defects within functions that are important to the business can be identified and resolved as early as possible in the lifecycle. This also drives business confidence up as quality expectations are met – what’s not to like?

The information obtained through testing, establishing technical risk versus business confidence, helps inform when “good enough” has been met and the product ready to ship.

It’s also a fact-based tool to inform a decision if “good enough” has not been met or the plan is off track to agree whether corrective actions are required. Or hard decisions must be made to adjust quality expectations!

Methodology Agnostic

Risk Based Delivery and Testing is not suited to one particular delivery methodology over another. While Agile delivery values the collaboration and communication which is necessary to enable a delivery team to define and assess risks to delivery, supporting a Risk Based approach, the same practices and principles can also be applied to a Waterfall or Iterative delivery.

In summary, an effective implementation of Risk Based Testing is only possible within Risk Based Delivery. The whole team must be aligned to the business outcomes being delivered, what constitutes “good enough” and the risks posed to meeting that quality threshold.

Don’t crash your plane!

If you’d like to learn more on how to obtain the Risk Intelligence to increase the success of your deliveries download our White Paper now.