We need metrics. We are asked for them often to give some numeric value for what we are working on. Why? Because numbers don’t lie.
Stakeholders want proof.
- Was the project successful?
- Do the results show the time invested was worth it?
- Does the pilot prove its worth a long-term investment?
Picking the right metric is a massive part of proving concepts, project success, and time/return results. There are specific systems in place already that allow us to pull metrics and measure change over time. The metrics for increasing sales are straightforward. If the project’s purpose is to increase sales of a product by 10%, we can easily document the sales at the start of the initiative, then measure sales after. If the sales increased by 10% or more, and the only change was the initiative, it is safe to say it was successful.
But what about initiatives that are not as direct? Say, I facilitated a workshop. In the end, stakeholders want to know was it worth it?
To prove this, I need to prove that the program showed a return on investment.
Ponzinomics by Robert L. FitzPatrick, author of False Profits, is the first comprehensive account of how "multi-level marketing" was created in America, escaped criminal and civil prosecution and spread all over the world. It is the first book to deeply investigate the multi-level marketing phenomenon and to fully explain how the legitimate business of "direct selling" was turned into deceptive pyramid recruiting.
It takes time to create that workshop and it takes time for learners to attend. This time takes attendees away from their normal duties. The salary paid for employees to attend a workshop instead of working on their normal job duties is at a cost to the stakeholder. Not only the salary of the attendees but also the time the attendees put their work on hold for a time.
Inadvertently, there is a loss of potential income or progress to the company by postponing work on projects that could have otherwise been worked on.
If that workshop fails, it could reduce future investment in training efforts. When this happens, you may hear feedback along the lines of, “I’m not sending my direct reports to that workshop because the last one was a waste of time.” or “We aren’t going to invest in creating a new workshop because they aren’t worth it.”
Let’s go through how we can show value. You may ask, was the knowledge was applied? Was the training functional? Will the participants attend another session? Does the team work more effectively? Etc, etc.
Often, when looking into how to capture these types of answers, there is a catch-all approach of “We can give a survey!”
Here is my controversial response: Surveys don’t measure success.
If you use surveys as the sole source to measure success, identify root problems, or decide what to invest resources in, you are doing it wrong.
I am not saying that surveys are useless. They can be instrumental when figuring out a population’s sentiment or identifying potential causes of a problem. They can be great for identifying a symptom of a larger problem, just not the success of a program.
Let’s go through a couple of examples to dig deeper into what I’m talking about.
Mary has been tasked with addressing the retention of graphic designers. Recently, the company has been losing talent to a competing company. Mary has seen this happen before and suspects that employees will be happy and stay if the company adds more support, perks, or team-building resources. Mary surveys to determine which perks the employees would most like added to their benefits. After providing a list of options, most of the employees chose an option to have weekly lunch provided by the company. So, Mary started a program to have lunch brought to the office every Friday from various local restaurants. After a few weeks, Mary polled the employees on their thoughts about the new lunch program and received overwhelmingly great feedback. After about six weeks into the program, Mary was very excited to discuss the success with her supervisor, only to be met with confusion. Mary was informed that not only was the company still losing graphic designers, but the rate of attrition was even higher than before.
Where did Mary go wrong?
There were several areas where Mary made some errors.
- She assumed a solution based on a past problem.
- The survey she gave was too narrow in scope and didn’t allow employees to identify the real reason for attrition.
- Mary determined the initiative was a success because the employees liked the free food.
- Mary didn’t focus on the true metric, the attrition rate of graphic designers.
The real problem was that graphic designers were leaving for a competing company. Suppose Mary had been more open to other solutions—she might have found out earlier that the root reason for the attrition rates was that a competing company was offering new hires a substantial signing bonus and the option to work from home. Although the Friday lunches Mary implemented were appreciated, it wasn’t enough to overcome why the talent decided to leave.
Mary was within her rights to try different things to increase the retention of the graphic designers. However, the only measure of success important in this initiative was not how employees felt, but how they behaved. The valid metric to measure success was if the company was keeping or losing its graphic designers.
The software engineering department has an excess budget at the end of a quarter. Engineering leadership has decided to use this budget to provide training to their software engineers to improve the overall productivity of the team. Bob has been tasked with determining the topic for this training that will have the most impact and collects the data on the average time to market for engineering projects. Bob decides that the best approach is to identify the areas the software engineers lack confidence. Bob is a trainer by trade and not a software engineer. He conducts surveys with several skills and topics so he can identify the knowledge deficit. He asks survey participants to rate their knowledge in the listed areas between 1-5, with 1 as no knowledge and 5 as expert knowledge. After surveying 1,000 software engineers, they self-identify “Machine Learning” as the least knowledgeable among them, with an average response of 1.3 in that topic. Bob creates an online course about “Machine Learning” and delivers the training to all 1,000 software engineers. At the end of the course, the participants are asked to rate their knowledge of the topic again. After a few weeks, Bob looks at the data and notices that the results are not good. Not only has the self-identified knowledge on the topic increased minimally to an average of 1.4, but the time to market for the software engineers has remained consistent.
Can we help Bob figure out why things went wrong?
Poor Bob! Let’s go through some of the reasons Bob’s training wasn’t effective.
- Although Bob did look at time to market, he incorrectly added topic options to his survey that did not impact this metric.
- Bob assumed that addressing a lack of conceptual knowledge would have the most impact on time to market.
- Bob made assumptions about what engineers interpreted as “no knowledge.”
- Bob relied only on his survey when determining the topic of the training program.
When digging deeper, we find that the software engineering team at Bob’s company was low in the knowledge of Machine Learning. However, only ten software engineers use machine learning in their jobs, so the topic wasn’t relevant to 990 participants. On top of that, Bob’s course covered basic information about what machine learning is and how it is sometimes used, but nothing on implementing it. It turns out, almost all of the engineers already knew the information provided in the course. They mostly marked the survey as “no knowledge” because they had never used it before. Because of this, they rated their knowledge almost the same as when they went into the course.
Bob could have gotten more information by seeking out more data points before creating his course. This could have included observing persons during their
For this initiative, Bob could have made the most impact on time to market by creating training around best practices and standardizing the troubleshooting process between the different software engineering teams.
One last example….
Richard just attended a conference where he learned that each time a company is featured in a “Happiest places to work”-type article, they receive an average of 25% more highly-qualified applicants for their open positions. Richard returns from the conference with a plan to make sure that the employees at his company are the happiest ever because attracting more qualified applicants is something the company has always struggled with. Although Richard’s employees have a great culture, Richard spends the next year investing in even more perks, amenities, and programs for employees. He surveys the employees often and documents the happiness ratings of his employees going up and up! But over the next year, the articles that rate company culture ignores Richard’s company.
What went wrong?
I added this one as a trick. On the surface, the obvious action is to invest in making employees happier. But, it wouldn’t be a trick if that was the case. In this instance, Richard’s employees were already very happy. They could have easily been one of the top companies on one of these lists if happiness was the only consideration. It wasn’t an issue of making the employees happier, it was a combination that the company wasn’t marketing itself enough to be considered for these types of articles and it wasn’t large enough to be compared with the other companies often on these types of lists. So, Richard invested a lot of time and money that resulted in happier employees, but not in any areas that would make the company article-worthy.
Surveys must be used as part of a holistic approach to research. They can be used to measure sentiment, get ahead of potential problems, or narrow down options or causes. However, it is extremely rare for surveys to be the measure of success! There is a big difference between saying, “Our courses are successful because 75% of participants indicate they would take another one of our courses.” and “75% of our participants enrolled in an additional course after participating in another course.” Only one of those measures can be attributed to success or failure.
Don’t get me wrong. I love surveys. They help us narrow infinite options to focus on the most likely options. Without them, we could be making assumptions left and right. Surveys are awesome when they are used correctly. They help with creating hypotheses before testing possible solutions to a problem. If you are conducting a survey and don’t know what problem you are trying to solve- that’s a problem. You don’t want to be a hammer only looking for nails. Make sure you are using the right tool that is going to help make results. Sometimes it will be a survey, sometimes not.
Ready for another twist in Richard’s story? There is an even more important metric Richard could have focused on: Can the company attract and maintain top talent! Who cares if Richard’s company gets tons of qualified applicants? The real measure is if he can convince those highly-qualified applicants to choose his company to work at over the others. In the long run, Richard’s company ended up doing okay in this respect. When he was able to get qualified applicants through the door, it was easy to convince them to choose Richard’s company over others because of its focus on happiness and wellness programs. Word got around about how awesome it was to work there and the company grew and grew. Until one day, it was big enough and well known enough to get on one of those lists.
But wait! There’s more! Was the real goal to be featured in an article? What was the problem that Richard was really want to solve? He wanted to attract more qualified applicants! If he would have kept his eye on that metric, perhaps he would have focused on some other initiative that would have had a bigger impact.
Comments? Feel free to comment below.
Questions? Email me at askburns@learnburns.com
If I feel that others may benefit, your question or use case may be featured in an Ask Burns article. If chosen, identifying information will be removed.
