Connie Goodwin: [00:00:00] Good afternoon everybody. Thank you for joining us today for measuring the ROI of CX at CIBC. So we’re gonna start with a little bit of audience participation if we could just after lunch, gotta get things moving, right? So how many of you. See yourselves in this. You know that improving customer experience matters, but you can’t quantify it.
Or you can estimate NPS improvements, Hey, if I do x, NPS will go up by two points, but that financial benefit feels a little bit murky. Or your stakeholders just need proof that what you are selling them is actually gonna move the needle. So how many of you see yourselves in one of these situations? All right, well, that’s what today is about.
So my name’s Connie Goodwin. I have the pleasure of leading Medallia’s Experience advisory team. We are a group of former customer or employee experience practitioners, and over the past three years, I have had the distinct pleasure of working with my friend [00:01:00] Andrew Gilling Water and the entire CIBC team.
You’ve heard them get accolades, you won Forrester’s Customer Obsessed Enterprise Award.
Andrew Gillingwater: Yep.
Connie Goodwin: The Medallia XP Award, all the awards. So congratulations on the work. Very, very well deserved.
Andrew Gillingwater: Thanks, Connie.
Connie Goodwin: Andrew leads customer experience governance at CIBC, and I’m thrilled to have him join me on the stage.
So, Andrew, you have been on a journey for the past couple of years of quantifying the impact of CX for your organization. So tell us a little bit about what inspired that journey.
Andrew Gillingwater: Happy to do that, and thanks so much for having us today. When I think about prioritizing cx, one of the things we did a few years ago that Steph talked about this morning is we did a maturity assessment, which you actually did for us, Connie, those few years ago.
And the number one thing that came out of that, we interviewed, I think 70 different executives, 200 partners, was we need an ability to understand the ROI of NPS. It was the number one issue that came out of it. So we stepped back and said, okay, well what do we need to do? What’s the problem we actually need to solve for?
And [00:02:00] if I think about CIBC, we really focus on making our client ambitions real. That’s our purpose. We live and breathe it every day. And with 14 million clients and over 20 million client sentiment data sources that our team looks at each year, whether it’s surveys, complaints, called transcripts, or other listening posts, there’s a lot of things you could prioritize.
There’s a lot of work that you could focus on to try to drive improvement. So we focused on four key things to help improve that. When we focused on discovery with our partners, these things came up over and over again. The first was cost benefit. What you mentioned before, you know how much it’s gonna cost to fix something.
You’ve got these 10 things to do, but what three do you wanna prioritize? What’s actually gonna drive the value, the benefit? So helping with cost benefit was really important. The second was quantification. Which of those 10 are gonna yield the most material impact, both from an NPS lift, but also a financial benefit.
So helping them quantify, and that really leads to business casing. If you wanna unlock funding, you need to be really crystal clear on the value it’s going to drive. And then for prioritization, that’s the number one problem our partners had, which is where do I start? What do I do, and [00:03:00] how do I prioritize work?
So those are kind of the overlying four things we wanted to focus on.
Connie Goodwin: Yeah, I think that’s really important because. You have to show the benefit of what you’re doing. Yes. And without that business cases just fall flat. So you’ve been on this journey to quantify this, to build this valuation framework. So tell us a little bit about how you approached it.
Andrew Gillingwater: Happy to do that. When I think about the journey we’ve been on, it wasn’t a clear path. It wasn’t a straight linear path. There was lots of different movement along the way, but we partnered with really great partners to help us get there. Our advanced analytics team, finance different lines of business, in particular the Medallia team, both advisory and the analytics team to help us.
So there’s four different things we did. The first two, I’m gonna talk to, Medallia, actually did the analysis for us and the latter two we did in-house because we wanted it to be something we could scale and we had lesson learned at that point. So the first one is driver simulators. This is foundational.
So for some of our larger programs, you think about digital banking centers, contact centers, we wanted to have driver simulators to estimate. If you ask [00:04:00] 15 questions in your digital survey, you have 15 different drivers. Which drivers have the most material impact to the overall performance of that program?
So that we could, for target setting, if we knew initiatives that were happening, we could effectively say, Hey, this driver’s gonna go up and down. We anticipate kind of the art over science. Estimate and forecast potential movement in NPS. So that was the first thing we did, and it’s a critical component of the valuation framework I’ll talk about today.
The second piece was operational analysis, and this was really pinpointing and correlating NPS to key operational metrics. So for us, we actually partnered with our contact center team and tested seven hypotheses to understand how does NPS move with first call resolution? So do clients actually believe you solved their problem at the first point of contact and we tested, you know, handle time.
Agent tenure, lots of different factors and it really helps you pinpoint what can move NPS, improve the experience, but more importantly also drive operational efficiency so your partners are speaking to the bottom line. You can reduce calls or the amount of time your clients are waiting on [00:05:00] calls they pinpointing where you can improve both.
That’s really important as a test and learn before you get to the next phase, which is financial linkage analysis. And what we did is we went through what I call A PIR post-implementation review, your lessons learned on that project, and we’ve thought about how do we collect the data? We thought about how do we engage our stakeholders before we went to financial linkage analysis to do this right, to get the buy-in across the organization.
So financial linkage analysis is essentially very similar to operational linkage, except that looking at financial measures, so revenue, attrition, or churn, depending on what you talk about it or other key financial metrics that your organization cares about, and you’re correlating that based on your passes, promoters and detractors, because we wanted to see do they behave in a similar fashion over time.
That is what you need to do. The last pillar, which we’re gonna talk about today, which is the valuation framework, and essentially what that does is it combines both your driver simulator so you can estimate the NPS movement with your financial linkage outcome. Once you have the data, here’s how much.
You know, promoters generate [00:06:00] revenue. Here’s their churn rates versus detractors, for example. You can use both to estimate your movement.
Connie Goodwin: Yeah. And this has been quite a journey that you’ve been on. I recall being in Toronto with you a couple of years ago where we started to build out this analytics agenda, and it’s, and it’s been a multi-year journey.
Mm-hmm.
Connie Goodwin: But we’ve gotta, we just gotta double click Andrew into the financial linkage analysis. So what did you test? What did you find?
Andrew Gillingwater: Sure, happy to do that. And you all say we don’t consider ourselves CX leaders. We consider ourselves business leaders that use client sentiment data to inform decisions.
And so we approached it this way with this work. What we actually looked at here is we made some assumptions upfront. We needed metrics if we wanted to scale this beyond the group we were working with and needed to be metrics that were channel agnostic. It wasn’t digital metrics, it wasn’t, you know, contact center metrics.
It was revenue that everyone cared about. It was attrition that everyone cared about. So we focused on what you see here on the screen. We looked at almost 500,000 surveys over a two year period across our digital contact center and banking center channels. And we combined it with a lot of operational [00:07:00] metrics and there was a few things we focused on, which is revenue attrition, funds under management, which for banking, it’s how much investments your clients have with you, assets, liabilities, your engagement in your relationship depth, which is essentially the products that you have.
So your credit cards, your accounts, as well as how often you use them. So when we measured this, we identified that our promoters generate about 15% more revenue over two year period. They attrite at a 4% less, um, rate of attrition. They have higher amounts of funds under management, more products, and they use them more, not just at the starting point, but they increased over time much more substantially than our passes and detractors.
And actually our detractors. We actually saw a decline in some of the. The metrics. So this was what we needed to go to our executive vice presidents, our senior vice presidents and all the different partners to say moving NPS does affect your bottom line. It matters. We’re able to get some buy-in.
Connie Goodwin: That’s incredibly critical here.
And so when you think about this, you are just laying the foundation for getting things prioritized because they move the needle.
Andrew Gillingwater: Absolutely.
Connie Goodwin: Not just because you [00:08:00] wanna do them, it’s because they actually move the needle. So what I’d love to know a little bit more is about this valuation framework.
Mm-hmm. And how you approached it.
Andrew Gillingwater: Happy to do that and you’ll probably hear me talk about this a few times today. Doing discovery and having been a project manager in the past in different lines of business, um, it’s really important to understand how decisions are made today, what data is being used today, who makes the decisions.
You need to understand that so you know you’re solving an actual problem. So that’s the first thing that we did, and through doing that, we knew we needed to build a tool that would help. With decisioning. So we call it a decision support tool to help with cost benefit analysis. And we didn’t want it to be something that was gonna consume a ton of capacity because we get pushback to say, Hey, if you know I got 10 things, I can’t spend an hour on each of those 10 things, my people are busy, they don’t have that.
So we needed to be simple and effective, a tool that they could use to quickly input a few pieces of detail to get the output. And we knew we needed it to be something they could consult with our team with. Because you likely need to have an understanding of how NPS is calculated, how sentiment data is used.
So this tool essentially does cost benefit. It [00:09:00] allows you, if you had 10 different initiatives, to compare those 10 to each other to say which of those 10 are gonna yield the most material benefit. You wanna get a million dollars to do three things, which three should you do? And this tool helps you do that.
Connie Goodwin: Yeah, that’s great. And so if I’m all of you, I’m going, this is nice, Sandra. Thank you very much. Can you show me, like, show me the money, show me what this looks like. So can you show us a little bit more about what you built?
Andrew Gillingwater: I hope so. Before we do that though, let me just recap where we’re at and then bear with me.
I’m gonna get into some nitty gritty of how the tool works. So the first thing you need to do, like I mentioned, is you need to quantify the ROI of NPS. Do passes per promoters and detractors behave a certain way. How much revenue do they generate? What do they trade at? And you need to understand that that’s imperative.
Once you’ve done that, I like to call it a simple plug and play tool, something that’s easy and effective that anyone could use that understands how NPS is calculated. There’s really four steps in it. The first is you anticipate the driver change. That is the driver simulator. The second is you need to understand for the problem or initiative you’re targeting, what’s the current state look like?
Proportion of the detractors, [00:10:00] promoters, detractors, and then the number of clients that project is going to impact. And then the tool calculates the associated financial benefit, whether it’s up or down movement for revenue, NPS and churn. So bear with me, I’m gonna walk you through the two tools and I’ve tried to remove our CIBC specific details here, but I’ll talk about it at a high level.
So the first page I’m gonna go through this one is our driver simulator, and I’ll click through each step so you can follow me through here. The first thing you need to understand is what’s the current state? So this is something that we actually get data for each year from a das Advance Advanced Analytics team.
I think they call it Multiline Regression. Bear with me.
Audience Question 1: Wow, that’s nerdy. Um,
Andrew Gillingwater: that is nerd special. They do it. And it helps you understand for all the different drivers in different programs which have the most material impact, and what is the current performance of each of them. So the first box you’re seeing here is, let’s just assume this is digital, and our digital program has a 60.5 NPS.
Then let’s assume driver A is ease. Driver B is, you know, I don’t know, effort, different drivers for your programs. What this is showing you is from all the surveys clients have done through that digital program, [00:11:00] what are the scores? So you can see driver A is 8.5, but driver B is 8.7. So you have clients that are saying, you’re doing really well in this area.
The second area is how much impact each of those drivers has. So we don’t show 15 different drivers in the tool. It’s too many. We show three to five that have the most material impact on the overall program NPS. And so, as you can see in this scenario, driver B has a more material impact overall than driver A.
So if you had a project that was going to affect driver B more, you’re gonna have a more positive or negative impact depending on how it’s going to happen. The next section is what I call the art over science, and this is where you would input what you think for your initiative. So let’s say you had a password reset initiative.
You’re gonna improve your authentication process. You could look at historical data, and this is what we do to say, do we have something similar happened in the past in this digital program that’s similar to this project? How did the drivers move? Did it go up or down? And by how much to estimate what we think this project is going to do so you have a more realistic gauge of what that is.
So in this scenario, we can see driver B is gonna [00:12:00] have a more material impact. It has a 0.4 impact, so it’s gonna have a higher lift. So for hypothetical situation here, let’s just say we’re gonna see a 1.2 lift, and that means the NPS is gonna go from 60.5 to 61.7. But I think it’s important to know it could also go down in working regulatory environments and or different initiatives.
They don’t. Always have a positive NPS movement. So this is important to know, to tell your business that everything you do is gonna help you move up. You need to balance that out when you’re forecasting the movements.
Connie Goodwin: Yeah, and this is where we go back to that pyramid slide that you showed a few slides ago.
Andrew Gillingwater: Yeah.
Connie Goodwin: Where this driver analysis, the key driver analysis was the foundation, and you needed to be able to do this to know which key driver is going to move NPS the most.
Audience Question 1: Absolutely.
Connie Goodwin: And what we see at Medallia is a lot of our clients stop here. And they say if we do X, we’re going to see NPS go up by 1.2 or we’ll see NPS go up by 0.5.
And unfortunately. That’s wonderful, but you can’t put NPS on the same shelf as you put revenue or [00:13:00] attrition or anything like that. So what we see is a lot of our clients stop here and you get frustrated. But NPS is gonna go up by 1.2. We should get this prioritized. Mm-hmm. But you don’t get it prioritized ’cause you don’t have the dollars and cents yet.
I’m looking at Boz here. You gotta speak the language of your CFO, which is. Dollars and cents.
Andrew Gillingwater: Yeah.
Connie Goodwin: So you’ve gone a step further.
Andrew Gillingwater: Yep.
Connie Goodwin: Tell us how you did that.
Andrew Gillingwater: Absolutely. It was really important for us, and having worked in the business for many years in hr, in different lines of business, you can’t talk only in NBS, you’ll get nowhere.
So we knew we needed to move a step further to talk about the bottom line impact. So I’ll walk you through a few of the steps here. This is a hypothetical scenario. So let’s assume you had a hundred thousand clients had experienced the password reset process. The first thing you would look at is the current state of that.
How many are passive detractors and promoters? So for this example, you can see 8% are detractors, 20% passive, and 72% promoter. That’s pretty good. But you wanna reduce your detractors. You wanna try to increase your promoters to improve the overall NPS. The next thing you wanna [00:14:00] do is from your analysis, when you do the ROI analysis, there’s different things in different cuts you can do in that data.
So you can look at different age ranges, segments, lines of business, what matters to your business. It’s gonna be relevant to you. So for us, we wanted to have relevant contextualized. Pieces into this tool. So age segment region. We have a few others, but I included this for reference. The reason that’s important when you do your analysis, one of the things we learned under funds under management, so how much assets and liabilities the clients have, we saw that our 65 plus year old clients who are having a very steep decline in FUM or funds under management.
That makes sense. You paid off your mortgage, you’re using your savings, you’re living your life, you’re pulling out of your assets. If you had a project that was targeting that group, you wanna account for that difference in the outcome because your estimate would be accurate if you didn’t. So you wanna have those nuance factors factored into your tool, whatever they happen to be from your research, to account for those differences.
For more realistic estimate. The next piece is actually pulls from the prior tool. This is built in Excel, so it’s very easy to use for our team members, and it pulls from the prior [00:15:00] tab. What the MPS movement is, it says, Hey, if you wanna move 1.2 for this project, it means you need to move about 1200 clients from the current state scenario.
You need to move those detractors out and into promoters, which is not easy to do. So that’s why we want this tool to help make those decisions. Then in the background, there’s a bunch of different formulas that we have here, but essentially it takes the output of the research that we’ve done. So, you know, let’s just assume for revenue, our promoters are in 3000 passes, or 2000 detractors or a thousand in revenue over a two year period.
That’s what they, they generate. It would take that into a factor along with all the other metrics, and it calculates what your lift would be. Now what we built is we actually built a range because we don’t wanna have a finite number and someone say, we didn’t hit that number. It doesn’t work. And so what we did is we know past behavior isn’t indicative of future behavior.
So we said, Hey, if it’s 3000, 2000, a thousand, let’s say we’re 80% accurate, so we did 2000, et cetera, et cetera. And then we built a shoulder upper and lower limit off of that to say. Probably not always gonna be a hundred percent on that number. So we’re gonna do [00:16:00] 20% lower and 20% higher to build a range.
And we did testing to validate the accuracy, which I’ll talk about that in a section. But what you can see here is that for this initiative, you could say between 20 to 50 clients and you could generate 700,000 to a million in incremental growth by doing this initiative, by generating promoters. And this is how we actually move the dial.
Connie Goodwin: What’s really interesting is like, this is, this is an example, but as an organization with 14 million clients, you can see the scale, like you’re giving a great example that this is gonna move revenue between 700 and a million dollars. But if this initiative is gonna cost you $300,000, yeah, you do it all day long, right?
And so scale matters here with an organization of your size, scale is really important, but in any organization, this upward and and lower limit, it just becomes a cost benefit analysis. If we spend 300,000. We’re gonna save this many clients and we’re gonna generate a couple of million to do that all day long.
This is what gets customer experience enhancing improvements in the same space as revenue generating improvements. [00:17:00] We all know our company companies get distracted by this shiny new toys because we say this will generate this much in sales, this much in revenue. This is how you put CX improvements on that same shelf.
So Andrew, this isn’t a. Field of dreams, if you build it, they will come. Situation. You had to navigate the organization, get organizational buy-in. So let’s pivot a little bit and talk not so much about the model, but how you worked within your organization to get buy-in to use this.
Andrew Gillingwater: Happy to do that, and what I’ll try to do today is share some of the approach lessons learned of what I would do differently.
Starting over, there’s three things you need to do. Of course, at the top, you need to measure the N-P-S-R-O-I, so you have the output to build the framework, and you need to build a tool that’s easy and simple to use that’s not super time consuming and can constrained capacity. So there’s three things.
The first is a vision. You need to be very clear on what you’re trying to solve and who you’re solving it for, and you’re actually going to have a [00:18:00] material benefit. The second is you need to work with allies or friendlies or however you wanna refer to them. People you have trust in buy-in across the organization that can help work for this and would benefit from the outcome of this work.
The third is you need a tool that can help you estimate both the NPS and financial benefit. That’s critical. Otherwise you can’t make better decisions. Yeah.
Connie Goodwin: And you being on the CX strategy team, obviously you’re gonna start with your strategy. You’re gonna start with your vision Of course. So as you were defining your vision, what became important for you?
Andrew Gillingwater: Now when I think about vision, there’s lots of different ways you could define this, but it’s really looking at how are decisions made today and how will your solution fix the gap that you’ve identified. So what’s the problem you’re looking to solve? How will you know you’re successful? And then what’s the scope of your work?
And I’ll touch on each of them and give examples of what we did. Defining the problem, like I mentioned before, during discovery and engaging with your different lines of business, your analysts, your consultants, your directors, senior directors, people who. Make decisions today or looking at data, you wanna go to them and understand.
So what we did is we actually had a list of questions for [00:19:00] discovery to understand what data are you using? Do you even look at sentiment data? How are you making decisions? Who do you send that data to? Is it in a report? Like we had a very thorough discovery process to identify the problem, and that’s where we identified it’s really the cost benefit and prioritization that they were struggling with.
Then we thought it to ourselves, okay, knowing that that’s the problem, what’s our unique value in the organization? What does our CX team do, and how can we help them support this? And how would we know that they’re successful in doing that? So success for us was, it’s not gonna be time consuming. It’s gonna be two, three minutes for them to input to this tool to do it.
So we’re not gonna get, Hey, I have no capacity to do this constraint. The second is. We needed to have a mindset shift. And if you think about the way projects work, you typically track your KPIs or your outcomes over 3, 6, 9 month period. You’re not tracking two years unless it’s a large initiative over time.
So we had to shift and help our organization know, you’re not just doing this. You’re not gonna move a tractor to passive, to a promoter, and all of a sudden you’ve generated a million dollars in revenue. You’re doing this because it’s the long game of getting more clients to be [00:20:00] promoters because they generate more value.
So that was success for us, helping people understand that, and this tool would help them do that. And lastly, scope. When we started doing the analysis, we socialized broadly to the executive vice president, senior vice presidents, all across the organization because we wanted them to know what we’ve done and the value that it would drive.
But as you do that, people are like, can you analyze this? What about this cut? Can you look at that? And we had to be like, no, we cannot get into analysis paralysis, scope creep. So being very clear on what your scope is, the metrics you’re measuring, and make sure your sponsor is aligned with that too, to say, this is what we’re doing.
This is, we’re gonna try to get this done, we can look at more later and iterate over time was really important.
Connie Goodwin: Yeah, and it’s interesting because I, I was involved in this three years ago and we did a, I think the beginning of it was the customer experience maturity assessment that we did. We talked to over 60 stakeholders in your organization, and what they said was, we want to be able to.
Quantify the, you know, how much revenue we’re going to get
Audience Question 2: Yeah. [00:21:00]
Connie Goodwin: From a CX improvement, but we, we can’t do it. Mm-hmm. And they were, they were really craving this financial linkage analysis this way to justify the spend.
Audience Question 2: Yeah.
Connie Goodwin: I mean, nobody has endless budgets, so you have to justify every dime that you’re spending.
So they were looking for this way to do it. So I think what you’ve given your organization is this valuation framework that allows them to. Spend the money they need to spend because it’s gonna going to generate the revenue that they think it’s gonna generate. So tell us a little bit about how you did that, how you approach this work with within CIBC.
Andrew Gillingwater: Happy to do that. I think Steph mentioned it today too. We interlock with all of our partners each year. That’s a really critical step to know what’s important to them. To also understand who you’ve built trust with, who’s listening to you more. You have a good gauge of who you’ve built that rapport with so far.
So when I think about buy-in, it’s really having a trusted partner to work with that’s also gonna realize the benefit. You have a mutual. Benefit, they’re gonna get value out of what you’re doing. So those four steps we took, the first is identify your allies. Like I [00:22:00] said, discovery. Once we had the group of people, we went and engaged, we went and talked to them to say, if we did something like this, would you be interested in it?
Like would you be able to benefit from that and to understand who would have the most material value. And then we developed a socialization plan, which is step two we built. Three slides that said, here’s the problem. We’ve identified through all the individuals we’ve engaged, here’s what we think we can build and what we want to do, and here’s what the impact would be if we got this done right.
And then we went and engaged the top two to three partners that we thought we’ve built rapport with. They’re an ally, we think we can work with them. And we actually identified the digital team as our partner. And it was really important for the digital team because if you think about how digital works.
There’s the sales component where you wanna get a credit card, you go through, you fill that online and you get a credit card to your house. But there’s also all the servicing elements. Your password, you’re trying to do transaction, pay a bill when it gets wrong. It’s frustrating, and it’s the servicing related initiatives that they have a large backlog with where it’s like, how, where do we start?
Where do we focus on? Because there’s so many things they can fix. And so once we had their buy-in, we collaborated with them to go to our advanced analytics team to say, we [00:23:00] wanna do this work, we want your help to do it. And that’s really important for the next step, which is you need to make sure you have a partner with a trusted voice and organization.
It’s the most important thing out of step two. Both digital and advanced analytics. Digital because from us where we want to win, being a digital first bank is really important to us. It’s one of the areas we wanna win. So we knew if we got it right, their voice would matter to our contact centers, our banking centers, other groups, they would sing the praise.
So that was important. But advanced analytics as well, because when we go to socialize the output, you don’t want to be the CX team that’s like, Hey, we did this. And they’re like. Did finance vet it? Did advanced analytics vet it? Like how do I know it actually works? We engage them at every step. Finance, review it every time.
Advanced analytics viewed every time digital analytics, our analytics to make sure when we got to the end, there was already trust in the output when we got to that stage.
Connie Goodwin: I think that’s incredibly important and, and I, it’s something that your team does remarkably well, is you are not. Pushing.
Andrew Gillingwater: No,
Connie Goodwin: you know, you’re not pushing this on anyone.
You’re finding your friendlies. I, that’s a term that you’ve [00:24:00] always, um, used and I’ve adopted. But you found your friendlies, they do the selling for you. Mm-hmm.
Andrew Gillingwater: That’s right.
Connie Goodwin: So now you’ve got your digital team saying, Hey, we’re doing this, and now your deposits team does it, and now your, your, um, cards team does it.
So you’re, you’re creating allies within your organization so you don’t have to go out and sell this. They’re coming to you, which is really remarkable. Tell us a little bit about how you got to this place.
Andrew Gillingwater: Sure, happy to do that. So there’s six steps that you need to take. The first is you need to understand the end user, who would actually use the tool, and how can you make it easy for them to use that?
So what type of work do they do today? What data are they looking at and what’s the processes they use to make decisions? So once you understand the end user, and in our case, we actually thought up front it would be the lines of business. We thought we would make it a self-serve tool, they could go and use it.
But as we got further down the line, we realized that’s actually not the right person. Our measurement team because they have a really strong understanding of historical data, NPS and sentiment, and we could collaborate and consult with them. We have regular [00:25:00] touchpoints every month with them, so it was something we could do.
We see our team being consultants and using the tool. After that, you need to figure out, you’ve done all these metrics, you’ve analyzed this, but you can’t have. 10 different outputs showing in this tool you need two or three that are gonna have the most material impact. So confirm the metrics you actually want in the valuation tool.
So revenue for sure was an easy one for us. Attrition and then funds under management. We want our clients to continue to engage with us and you know, get more, more engagement with us over time. After you’ve done that and you kind of have the base scope of what you need, you need to develop a prototype.
So we built it in Excel, and the reason we built it in Excel is it’s something accessible across the organization. Everyone has access to it and it’s very easy to use. We built the prototype. We didn’t have every single data point in every analytics in the backend complete. So I mentioned we had the 65 plus, but for some of the other demographics or the other areas, we didn’t have a lot of data.
But rather than go back and do it again, we built the prototype with what we have to test it. So we built it and we did retrospective testing. And I would recommend you do this. So retrospective testing or historical testing is [00:26:00] pretending you had this tool two years ago and picking an initiative that launched where you can actually track the movement and those changes over time.
So we picked password reset, um, and the reason I feel like I keep picking it up. The reason we picked that tool is it’s something that you do often. Clients experience pass through, reset more than once. And we could track the change in NPS pre post and we could track the movement in financial values over time.
And once you’re doing that testing, you can validate your accuracy. So I mentioned the upper and lower limits. That’s something that we wanted to validate. Did we actually have the right limits? Were it too high, too conservative or not? And adjust them. And then you refine your framework based on that testing.
Now the testing’s really important too, because you want to be able to get to a point. You’ve done the analytics, you built this tool, but you’re still gonna have skeptics, and you’re gonna still have people that say, is this realistic? Does it actually estimate or forecast the future? Well, now you have a proof point.
We’ve did this, we’ve added it, vetted. We’ve done this, we’ve had it vetted, and now you have a proof point. Password reset did this. Here’s what the outcome was. It works. Let’s go use it on real scenarios now and estimate moving forward.
Connie Goodwin: I [00:27:00] think that’s really remarkable. So one of the things, Andrew, that you know, I’m, I’m very curious about is if you could give this group your lessons learned in all of this, you know, done a lot, got great organizational awareness, built this, this great tool, what three pieces about.
Pieces of advice would you give to this audience?
Andrew Gillingwater: It goes back to the three steps I mentioned before. It’s being very crystal clear. What is the gap? What is the problem you’re trying to solve? And how are decisions made today? How will that actually help you make better decisions in the future? Buy-in.
You need a voice that matters, and some of that’s gonna get material benefit from the work that you’re doing. So they have buy-in and they’re willing to advocate for you if it’s done well and effective. And third is you really, you need a simple tool that helps you in a very. Easy way. Estimate the uplift for NPS and financial benefits.
Don’t just talk in terms of CX metrics, but talk in terms of it is project that you want to do. It’s gonna yield X value, and here’s what the input is gonna be overall to the organization.
Connie Goodwin: If we’re [00:28:00] talking NPS, that’s great. NPS is important. Um, I mean, I’m a practitioner for crying out loud. NPS is important, but it’s only as important as the business results that it generates.
And so we have to be speaking the language of our stakeholders day in and day out. So Andrew, thank you for sharing this. We’d love to open it up for questions. And while we do that, I mean, hello. We are a feedback organization, a customer experience and employee experience organization. We’d love your feedback on this session so that we can learn from it.
So if you could, you know, we’d love your feedback, but why don’t we open up for questions from all of you.
Audience Question 3: So my, my question is, when it comes to your, the simulation that you mentioned, could you elaborate on like the type of data that you were, uh, pulling from, I think you mentioned your, uh, advanced analytics team.
Andrew Gillingwater: Yeah, so there’s two pieces. So if you’re talking about the driver simulator, that’s using the MPS data that we have in medal. So we work with me’s, advanced analytics team like Roche and team who helped to do that. And for our key programs, it’s actually our measurement team that leads that work. Um, we analyze for digital contact centers, banking [00:29:00] centers.
We look at all the drivers to understand the movement of those patterns based on all the surveys that are completed. For the second piece of work, the financial linkage analysis, we like to look at transactional, behavioral and sentiment data. You could refer to operational for transactional, depends on your organization.
Uh, but we looked at all of those elements, so we needed to know who the clients were. We needed to know a bit more of the details about those clients, who they are, their age, their tenure with the organization, some of those key factors. And then we started first with the surveys, so the almost 500,000 surveys.
And then we layered in the additional metrics to that. So all of the operational metrics about the clients, all the funds under our management. And rather than do it at, um, an individual level, we had aggregated some of it. So we had ranges, which is like your funds under management was 25,000 to 50 or 50 to 75,000 to get that to help get approvals to move faster from a risk review standpoint.
Um, and we combined all of that data, did the analytics, and we did a lot of testing. So it wasn’t like you did it, you were done, we were good to go. I think we did it three, maybe four times where we had to go back, do further analysis, analyze it a little bit further. [00:30:00] So I don’t know if that answers your question, but I would start at the survey data and then I would look at.
Adding the client attributes to it as well, that matter. So revenue, obviously, you need the revenue figures to churn all of those metrics.
Connie Goodwin: The other thing that I would add to that, Andrew, is that this has been a, a true partnership over the years where the analytics that we’ve done, I, I remember being in Toronto a couple of years ago saying, we need to build your analytics agenda.
Audience Question 2: Yeah.
Connie Goodwin: And this is something that with the driver simulators, Medallia handled. And then when it came to the operational linkage analysis, we did it in partnership. And then when it came time for the financial linkage analysis, you said, all right, we’re good. Yeah, we got it. We’ll consult with you. So this type of partnership is one that all of you can do.
You can do it on your own, we can do it for you. Mm-hmm. Or we can do it in collaboration. And I think that’s probably the best, the, the true collaboration where. You leverage people like Ro and Noam who presented earlier and our, our wonderful Medallia strategy and analytics team, and then your own resources.
So if you have the resources to do it yourselves, you can do it. If you [00:31:00] don’t, we can do it. So it, it really is a great partnership.
Andrew Gillingwater: Yeah, and I think to add on to that too, I think our measurement team has done really well and they manage all of our programs. We have over 50 NPS programs that we run. We brought in a whole bunch of operational data into Medallia.
So for each survey, we have a lot of details about the client’s, the hierarchy structure, everywhere. They line up to that in order to do different analysis. So we exported the files to do the analytics. We had a lot of what we needed. It allowed us to speed up our process. And to your point. The reason we did the first two with Medallia is we wanted the outside in perspective so that we could avoid sharp corners, we could move quickly, learn fast, iterate.
For the last two, we built it in-house because we wanted to be able to scale it beyond just digital. We wanted to scale it to other lines of business and be able to manage that model in-house. Uh, but we did end up funding a resource in advance analytics to help with maintenance and some more advanced analytics We’re doing.
Connie Goodwin: I think that’s an important part is that you, you poi up an ft.
Andrew Gillingwater: Yeah, we
Connie Goodwin: moved into the analytics team and said we need someone to do this. So I think that was an important part.
Audience Question 4: You mentioned that you kind of, [00:32:00] in, as a part of your model, you looked at where previous programs had impacted MBS. Can you talk about how you did that?
Did you just look at like, I know that this went live on this day, Lexi, like the before and after, or like, was there a specific framework around doing that? Did you look specifically at. Like comments around, let’s say like the password reset. You mentioned or like how did you do that?
Andrew Gillingwater: So the first piece of work is we actually worked with, I dunno if Roche’s in the room, if she could put her Roche’s there.
Um, we actually worked with her team to do the Multiline regression. Um, and that was really impactful for us. They actually get refreshed each year for us because as you can imagine, it’s not static. Human behavior is something you need to monitor over time and you might have had a mean score of A, B, C, but next year it’s completely different because the amount of.
Times clients have responded to you. Mm-hmm. Ease might have gone down and effort might have gone up or vice versa or things like that. So they actually do the analysis. So if you’re looking at that, I talked to them, they’re the experts on that. Like I said, I’m not an analyst, but we worked with them to do that.
If you’re talking [00:33:00] about the part where we looked at, um, the password reset for the testing, what we looked at, we actually went back and looked at historical data. So we looked at a six month. Prior to the password reset six months post. And then we looked two years later. And the reason we did that is if you think about recency, if you’re looking at two years later of an NPS survey, they’re not gonna remember the S reset two years ago.
So we looked at the NPS movement in that short period of window, people who had pre and post, and then we had enough sample in order to do that. And then we tracked the change in outcomes in financial metrics at key points. So six months a year, two years later to see the incremental benefit. And that’s where for the long term game that I mentioned with your.
Analyzing and work with your project delivery teams. The people who are probably gonna measure if there’s, in a business case, this metric, you need them to understand. You’re not gonna see the immediate benefit, but there’ll be early signals. NPS might start to move. You might see some incremental funds being moved, or you might see some increased account activity.
Those are early signals that you’ll yield. Full two year benefit that you’ve estimated, but they need to [00:34:00] understand that process and how it works, which is why we’re the consultants on the project and we’re not just alone letting them go and do it on their own.
Connie Goodwin: The other thing that I would say is, I don’t know if anybody attended the session yesterday with Santander, they did something very similar.
When they, when they go and they try to get things prioritized, they come with the financials. They don’t try to put. NPS on the same shelf as revenue and attrition. They recognize that I’ve gotta talk the language of my CFO. I’ve gotta talk the language of my business partners. They’re talking revenue, they’re talking attrition, they’re talking household growth, they’re talking things like that.
What Santander did is the exact same thing that they made sure that they looked at lookalike initiatives. Yeah. Here’s what we saw on the last time we made this fix. We should expect to see something similar. So that was something that was very helpful for them. I dunno if that’s helpful for you.
Audience Question 3: So, uh, if I could revisit that point you mentioned where you use that NPS data to estimate [00:35:00] the financial, uh, benefit that you receive from the increase, how can you be sure that you attribute that financial benefit only to the NPS score and not to other noise?
Andrew Gillingwater: Yeah. One of the things I’d say is you cannot normalize for all conditions. Anyone that’s tried to do causation or correlation analysis, it’s very challenging to normalize for everything that goes on. So we weren’t the only experts in this. Like I said, the important piece for that is we had finance involved in every step and they.
They took us to talent on really digging into each piece, and we went back and did Rea Analytics to, based on their questions and their feedback, advanced analytics helped do the work. So they’re experts in the data and they also have a voice that when it’s done, people trust the output in that. So the way that we did that was we leveraged the expertise we had in house with the partners to do that.
We’re not experts in how the revenue data is calculated, how it’s working each month, the data lakehouse in every area, and we don’t have access to every single data point. So we needed to leverage the expertise. So my advice would be go to the teams that have the expertise, bring them along the journey to educate you, to [00:36:00] help you test and stress, test it.
And the other piece that’s important is your skeptics. And I’ll mention even our own CX analytics team was very skeptical. They were like, we shouldn’t do this. It’s not gonna work. They did it at another organization before they started, and I was like. I got this, we, we could do this. We had a vision. We knew if we did discovery, we could get it done right, but we engaged that team along the way.
So you don’t ignore your skeptics, you purposely engage them because you wanna understand why are you skeptical? And you wanna understand throughout the process, Hey, we’ve done this, does this make sense? Or like, what really? Like if you were to question us, go to town, what does it make sense to you? And help us get better.
So bringing them along the journey got us to this point. We wouldn’t have been as successful had we not engaged those teams in the skeptics along the way.
Connie Goodwin: You worked with Roche and, and our Medallia strategy and analytics team. You worked with your advanced analytics team. You found your friendlies, you found your skeptics.
So this was a true organizational, uh, initiative. And I think that’s one of the things that made it successful is invite. Get your friendlies, get your fans, but [00:37:00] get the skeptics. Let them pressure test it because it’s only going to make your model your positioning stronger. So I think you’ve all done a remarkable job there.
Audience Question 2: We’re a big bank, so we have a lot of different stakeholders and everybody defines ROI differently. This was just one definition of what ROI means, but the wealth team will have a very different definition of ROI versus the credit card team. So if you need to tailor your ROI definition. Based on the business that you’re supporting, like do that right.
There’s no one definition for ROI and I just wanted to make sure that everybody heard that here.
Audience Question 1: Thanks so much. Uh uh, my name’s Austin. I’m a product manager at Capital One. Um, I work in our strategy division and so from like that perspective, it sounds like the takeaways I’m getting is like. Sort of an amazing win for like influencing leadership and like connecting the dots and saying, Hey, NPS.
I think some people see it as an output. In this case, it’s really an input to ROI. Yeah, so I guess I’m curious, given the [00:38:00] success and sort of like naysayers and people saying, no, no, no, you’ll never succeed, and then you best them and you show them that you can totally do this. Has this changed your team’s roadmap?
Has this changed how you see yourself strategically in the organization and like, are you sort of changing how like you guys do CX given something? So I guess, uh, moving.
Andrew Gillingwater: So I lead our strategy and our roadmap. So what Steph talked about, Steph and I built, uh, the last four years together and we’ve been delivering on it with the data ecosystem.
And this work, this was part of our roadmap. Same with awards and speak engagements. There was a lot of different factors to that to help drive a mature organization. This didn’t change how we approached, but it elevated how we made decisions. So our roadmap was always focused on client-centric, data-driven decisions.
So everything we’ve been doing for the last four years has been to enable that, bringing the data together for the ecosystem, leveraging that ecosystem to connect to key platforms like our CRM platform for our frontline team members, our marketing data, Lakehouse, to make better decisions and personalized engagements to clients.
And another key pillar and enabler to this was the financial linkage analysis. And there’s more work [00:39:00] to come. We’re doing synthetic NPS, we’re trying to anticipate NPS, this project we’ve. We’re overseeing at are in flight with right now that will help us with this work because you’ll no longer need to guess.
You’ll know the sentiment of all clients at scale and it will constantly evolve over time and we’ll be honing that model that will help with this because when we do testing each year, we’ll have sample, we’ll have lookalikes to look at for this. But what I’d say is it didn’t change, it just elevated the way that we’re making decisions today.
But this is the goal where we’ve been on and continue to focus, being really crystal clear where you wanna win. Where you wanna compete is probably the foundation to start there and then figure out the steps you need to do to get there.
Connie Goodwin: Prior to joining me, I led client experience at a regional bank in the us and uh, I recall going to our CFO at the time and said, look, if we do these eight things, we will generate X in revenue.
C CFO said, I’m sorry, what? I, I, I thought CX was NPS and warm and fuzzy. We should just be nice to clients. I said, no, CX is enable of business outcomes. [00:40:00] And we showed the analysis that we did, and all of a sudden it was like, well, you just have a business case, so let’s just go do these things. And so when you start speaking the language of your stakeholders, the language of your CFOs, that’s when you really get momentum.
Because NPS, it’s wonderful. It really is. It’s gotten us to where we are, but it, it can’t be on the same shelf. As revenue churn, all of those business outcomes. So when you start speaking the language of your stakeholders, and even in digital, it can just be digital containment, right? Whatever. Whatever those goals of your stakeholders are, when you can start speaking that same language.
That’s when the unlock happens, right? And this is a team that considers themselves not just a CX team, but a true leadership team. They have a seat at the table and they’ve earned it through doing this type of work. So again, all the accolades that have come this, the, the way of this team are incredibly well deserved.
Thank you for joining us. Really appreciate, um, [00:41:00] all that you’ve offered to this group. And, uh, again, congratulations on all the accolades that have come your way and all of you. Thank you so much.
