How the test-and-learn process supports business innovation

Data can serve as a window into customer preferences and needs. However, many companies lack the ability to extract meaningful insights from their data at scale. Businesses know they need to make more informed decisions but may not know how. Enter the test-and-learn process.

Just as a scientist develops hypotheses and collects data to assess the validity of their assumptions, leading companies engage in small-scale “business experiments” to make more informed, data-driven decisions. 

When businesses introduce testing and learning to their strategic decision-making, they can cut through noise and ambiguity to identify and act on the highest ROI initiatives possible. With this rigorous testing method, scientifically sound experiments reveal the overall incremental impact of an initiative, the most effective aspects of that initiative, the types of customers or markets that respond best and how to target rollout across the organization.

Testing ultimately accelerates innovation by enabling businesses to rapidly explore ideas within a controlled environment without the risk of rolling out an ineffective program to the entire organization.

It can be challenging to identify the true business impact of a new initiative; however, the test-and-learn process helps companies understand what works best and why. Before committing to a full rollout, an intentionally crafted business experiment can reveal:

• The impact on key performance metrics
• How to tailor and target an initiative for maximum effectiveness

Based on a Mastercard Test & Learn® client survey, 44% of business initiatives don’t break even. Used by many leading brands, in-market testing empowers business leaders to improve the success rate of their initiatives and understand their overall incremental impact. By mitigating certain risks, companies can feel more confident in their business strategies.

In short, the test-and-learn process helps organizations maximize learnings, minimize risks and stimulate innovation.

The test-and-learn process involves implementing a change for select test groups and withholding that change for matched control groups. For example, a financial institution could send a credit card promotion to select customers only, or a retailer could implement a new store layout in some locations, but not all.  

Test versus control experimentation is the analytic gold standard for determining casual relationships. Other analytic methods (e.g., regression analyses) are helpful hypothesis generators, but only a business experiment that observes actual performance can isolate the true causes and effects of an initiative. 

In an uncontrolled environment, there are unlimited external factors that can affect performance, making it extremely difficult to isolate the true incremental impact of an action. An internal Mastercard analysis found that 45% of campaigns have natural test vs. control bias, meaning there is great room for improvement in analysis structuring.

Below is a walkthrough of the test-and-learn process, from hypothesis to execution.

The process always begins with creating a testable hypothesis.

Example: A retailer believes that a change to store layout will drive customers to visit more frequently, thus increasing sales.

• A hypothesis must be testable and have realistic success criteria that are defined and measurable.
• It should be possible to pass or fail the test based on the measured goals of the hypothesis.
• Stakeholders should align internally on when a test is necessary and when it is unnecessary.

• Does the experiment have a clear purpose?
• Is the experiment doable?
• Have stakeholders committed to abide by the results?
• How can we ensure reliable results?
• How do we get the most value out of the experiment? 

This step ensures that the test is properly designed and variables are identified in advance. 

Example: The retailer identifies highly similar “control” sites where the store layout change didn’t occur and uses them as a performance baseline.

• Defining test variables and success criteria ensures conclusive learning and positions the business to be able to move forward with a decision. The test should be set up to address the business goals and minimize bias.
  
  
• Appropriate definition of the test’s size and scope can generate more accurate results. Businesses should make sure they have:
  • A large enough pool to form robust test and control groups
  • An appropriate test duration to overcome random noise
  • A finalized test schedule to avoid overlap with other initiatives