Multiple Metrics for Measuring Chatbots

Originally published in Advertising Age
About 20 years ago, marketers started adopting metrics for online advertising that remain commonplace today. About 10 years ago, The same thing happened when the advent of smartphones spurred the rise of mobile marketing. As conversational commerce takes off, this is happening again because of automated conversational software known as chatbots, or “bots.”
Below are some approaches to consider for bot measurement. Note that little of this is currently available from chatbot platforms such as Facebook Messenger, Kik, Slack, or Amazon Echo. Bot analytics and bot-building software companies all have shortcomings, largely because this technology is in its infancy. Few benchmarks exist, especially when trying to compare data across platforms.
Marketers must start somewhere though, so here are a variety of metrics to consider:
Reach: Every marketer will want to know how many people are using their chatbots. Without further context though, this metric means nothing. Marketers should at least distinguish between active and inactive users, and break out active users by different spans such as the past week and month.
Average session duration: How does one define a chatbot session? A weather bot may define sessions as lasting for a few minutes, as someone would probably check the weather and then leave. A role-playing game may opt for much longer session windows. This matters in reporting. If someone starts shopping within a bot, and then resumes that process an hour later, some reports may say that’s one session, and some reports may say that’s two.
Frequency (or the number of sessions per user): This is also complicated with bots. Consider a bot from a chain of coffee shops. If someone visits it every weekday and inputs an order, that could be a frequency of five in a week. What if, instead, that bot sends daily push notifications with each day’s special flavor, but the customer never interacts with the bot? Is that also a frequency of five? It will also matter if the alert includes some of the message, or if the message must be tapped in order to be read.
These questions lead to another challenging topic: how do you know if your result is good or bad? For instance, the marketer behind the coffee bot would probably love lots of short sessions, every day. For a bot focused on customer service, one would hope that problems are solved quickly, and that the frequency is close to one. A bot developer pushing out periodic videos may optimize around lower frequency but longer duration.
Branding: What will branding studies look like for bots? What if consumers are asked a question and then must select an emoji that correlates with brand favorability or purchase intent? As more bots pass the million-user milestone, valid brand studies will become increasingly in demand.
Research: Much of the value of chatbots can come from harnessing data. For instance, when offering a menu with multiple options or a gallery of images, marketers can review which response draws more interactions. Marketers can also test copy through mechanisms such as randomizing the bot’s responses and determining which one delivers the best results.
Consumer contributions: Bots can solicit ideas or content, so they can be part of a consumer-generated media campaign. Content typically goes directly to the bot publisher, rather than getting published publicly, so the marketer can monitor everything submitted.
Attrition: What’s the churn rate over time and after each notification? The best reporting will show not just when consumers remove a bot, but when they mute it. Muting a bot stops notifications but keeps the bot in the user’s feed, so muting could be a sign that the content is good but the alerts are too aggressive.
Cannibalization: As a bot’s use scales, is it driven by acquiring or retaining consumers? Right now, most bot use is probably additive rather than cannibalistic, but over time bots should replace some push marketing (e.g., app alerts) and pull marketing (e.g., search).
With their first forays into building bots, most marketers will be underwhelmed with the metrics available. Nevertheless, marketers building bots should urge partners to prioritize reporting on the metrics that matter most. This is the moment to shape the future, as metrics that become standardized over the coming year could stick around for decades.
Total interactions, and interactions per user: It may be good when customers are expending a lot of effort interacting with bots, or it may be terrible. Perhaps users are interacting too many times because the bot doesn’t respond intuitively. Analytics tool Botanalytics further segments interactions into “conversation steps,” with each step defined as a single exchange between a user and a bot.
Conversation sentiment: People act pretty weird around bots. Some people profess their love to bots, some curse at THEM, and others do both. Basic sentiment reporting alone is useless without analysis. When people say, “I love you” or “I hate you” to a bot, is it because the bot made the person feel this way, or IS the user just playing games?
Confusion triggers: This term arose in the bot developer community for how often bots return a non-response such as “I don’t know.” The goal is to learn from each dead end and minimize the odds of it happening again.
Click-through rate: CTR metrics will surface on bots. Clicks are most likely positive indicators from consumers, but clicks could be a sign of failure if the bot experience is supposed to be self-contained.
Lead generation: Bots can be used to drive newsletter signups, meeting appointments, downloads and other conversion-related activities. For now, these are largely two-step processes, with users needing to click to a site and complete the task. That is likely to change as bots become more sophisticated.
Sales: What will people buy from bots? The user experience is challenging, as bots present linear feeds with little room for visuals. Attribution will be yet another challenge, as bots will most often lead customers to a website or a physical store.