Risk Evaluation of a New Payment Method
Although this case study is based on a real project, all data and variables have been simulated to ensure confidentiality
An e-commerce company was considering a new payment method which, despite its growing popularity, faced social responsibility controversies. We conducted an experimental evaluation to assess its impact on perceived ease of use, perceived brand responsibility, purchase intent, and customer loyalty. The results supported a recommendation against adoption, predicting a drop of up to 2% in customer loyalty without significant offsetting benefits.
BEFORE
Consideration Based on Popularity
The initial proposal of including the payment method was driven by external market trends
AFTER
Decision Based on Customer Data
The final recommendation was driven by experimental data on customer predicted behavior.
Project Overview
-
Duration: 2 weeks.
-
Role: Eduardo González led the project as the UX Researcher.
-
Team: Cross-functional team including Product Design, Research, and Product Management.
-
Key Tools: R (for data analysis & modeling) and Qualtrics (for survey design).
-
Full Analysis and Code: Available on RPubs.
-
Research Question: What is the impact of adding the payment method on customers' perceptions and buying behaviors?
-
Methods: Experimental study with 800 participants comparing the inclusion or not of the payment method on validated measures of ease peceptions, social responsability perceptions, purchase intention, and loyalty.
Planning a Holistic View
An important challenge was evaluating the multifaceted impact of the new payment method. To achieve this, we designed an experiment that:
​
-
Included diverse customer segments: We recruited both loyal brand customers—hypothesized to be more sensitive to the payment method's ethical controversies—and potential new customers.
​
-
Went beyond typical metrics: We moved past the standard "intention to use" measure. We focused on key business drivers: perceptions of payment ease, perception of brand social responsibility, purchase intention, and customer loyalty.
Note. Customers profiles were randomly assigned to a checkout page with or without the new payment method.
Rigurous Statistics to Detect Small Effects
Given the company's scale of millions of customers, even a minor shift in brand perception or loyalty could translate into substantial financial impact. Although we had a robust sample of 800 participants, traditional frequentist methods (like ANOVA) lacked the sensitivity to isolate small effects.
​
To ensure this sensitivity, we employed Bayesian analysis with multiple priors. This approach successfully identified a small but reliable negative impact on the perception of the brand's Social Responsibility. Furthermore, path analysis revealed that this negative perception was directly linked to lower Customer Loyalty, uncovering a hidden risk to the company's sustainable growth.
Note. The path analysis was crucial to distinguish between chance variability and theoretically grounded effects.
Expressing Results in Actionable Metrics
To ensure our findings were directly actionable, we expressed the results as the predicted percentage of customers who would change their perceptions or behaviors if the payment method were introduced. This approach facilitated the translation of customer results into business goals.
Note. Predicted drop in Brand Responsibility and Customer Loyalty, the only two metrics that showed significant effects.
Impact Evaluation
Risk Prevention
Prevented an implementation projected to erode customer loyalty by up to 2% without offsetting benefits.
Strategic Impact
The findings informed key decisions across 2 executive strategic planning meetings.
