Happily quantifying employee usage of Copilot (image source)
Across dozens of conversations with clients about Microsoft Copilot for Microsoft 365, I’ve heard a consistent critique: Data reporting has been lacking. As one client told me in mid-February: “The telemetry data isn’t there yet. We need more than just a 30-day look-back of utilization. We need trending prompts, use cases, and hard-dollar ROI.”
Aside from that hard-dollar ROI (the elusive holy grail of generative AI), Microsoft’s new Copilot Dashboard aims to significantly upgrade the data available to Copilot M365 users. Embedded within Viva Insights — users without a license can access some features through the M365 admin center, but the full experience requires a Viva Insights subscription — the new Microsoft Copilot Dashboard offers insights such as:
- Adoption statistics by group and by app. Leaders need to know how many users with licenses are actually using Copilot and in which applications. For example, teams may differ in adoption rates due to variations in the kinds of tasks employees perform or differences in their underlying AIQ. Within the M365 suite, Copilot applies differently to Teams, Outlook, Word, PowerPoint, and Excel. By knowing the usage rates, you could tailor the training you offer by specific apps.
- Before and after Copilot comparisons. The Dashboard can tell you whether Copilot users are spending less time in meetings than non-users — the supposition being that they’re relying on Copilot summaries instead of attending. We have some reservations about this scenario because we don’t think that generative AI (genAI) is trustworthy enough to summarize meeting outcomes without some further exploration on the part of the human user. But Copilot Dashboard can help you quantify if there are impacts on meeting attendance, plus less controversial statistics like the number of meetings employees have summarized.
- Employee sentiment analysis. You can survey your Copilot user base and compare their results to a global survey conducted by Microsoft. You can learn how they feel about the quality of outputs, the effort saved by using Copilot, speed of getting work done, and impact on productivity. It will add to your arsenal of employee listening efforts.
What’s Next: Quantify Progress And Train Employees
With the Copilot Dashboard being so new, I haven’t yet spoken with many clients using it. I’ll be keeping close tabs on how it performs and how well it informs clients’ workforce genAI strategies. Right now, leaders need to keep in mind that:
- Third-party partners fill in some gaps that Copilot Dashboard doesn’t cover. Security services vendor Varonis partners closely with Microsoft on services for Copilot. Varonis offers a suite of services designed to shore up a company’s security posture and data hygiene before deploying Copilot for M365 but also offers deep analytics capabilities. Varonis’ solution monitors Copilot data access in real time and detects abnormal Copilot interactions. For example, Varonis can audit prompts that users have entered into Copilot and identify those with malign intent, or those without malign intent that yielded undesirable outcomes.
- Hard dollars aren’t easily measured right now. In our foundational report, Build A Business Case For Microsoft 365 Copilot, we showed how the return on investment that you can calculate today doesn’t involve dollars or euros in your pocket but rather a return on labor time-savings. We’ve updated our core business case a bit since then, but the measurement challenge remains. The vast majority of Copilot customers can’t quantify the hard-dollar ROI of usage yet.
- Quantification is key to deepening adoption. Even if you can’t yet calculate that elusive hard-dollar ROI, you can drive numerous business benefits through quantification, like tool adoption and user productivity improvements. Which employee groups use Copilot? How often? For what apps and tasks? And what’s their feedback? All of these numbers feed into an employee experience-centric view of Copilot that puts the humans at the center of the analysis.
- Lots of training lies ahead. In my recent vision report, How To Drive Employee Productivity With Generative AI, I make the bold call that companies are on the verge of underinvesting in employee training on genAI by an order of magnitude. Microsoft aims to help with tools like its Copilot Success Kit. But as our major research stream on AIQ (the artificial intelligence quotient) demonstrates, employees lack the understanding, hard and soft skills, and ethical awareness to be truly effective and productive with generative AI, including Copilot for M365. Copilot Dashboard won’t solve these problems presently. You can start the journey toward AIQ upskilling with our reports, Prepare Your Entire Workforce For AI Now and The Forrester Artificial Intelligence Quotient (AIQ) Assessment.
J. P. Gownder is a vice president and principal analyst on Forrester’s Future of Work team. Clients can request a guidance session with him to discuss generative AI, including Microsoft Copilot for Microsoft 365 and other genAI employee solutions, employee productivity, employee collaboration, and other topics.
