Using Copilot Enterprise in a Teams call to summarise the data gave two different responses. The first was me free-form asking it to tell me what we'd talked about. This response seemed okay until the end where there references to customers / projects that weren't discussed (and I'm not even sure are customers!). The second request was from clicking a built-in prompt. This produced a much more concise and accurate summary.
Takeaway: Microsoft must be applying some massaging in the background with how to format & present back data from it's curated Copilot prompts.
Background
Copilot is a catch all name for Microsoft's implementation of Large Language Model (LLM) AI implementations. It is currently based around the Open-AI GPT engines and other associated technologies.
Generative AI is currently a very hot topic and most companies are interested to see how this burgeoning technology can help improve their business processes. An issue we may find with this type of AI is that you have to be careful about what and how you ask it for information to ensure that you get back both accurate, and comprehensive answers. In this style of post I am recording interactions that I've had with Copilot to chart and document what types of interactions are successful and which may not be.
Find more in this series here:
Interaction Summary
Technology: Copilot Enterprise inside Microsoft Teams, started during a 1:1 call
Data source: Teams call transcript (~20 minutes of discussion)
Date: Feb 2024
Prompts:
- Summarise what we talked about
- free-form prompt
- Summarize call so far
- offered prompt, selected
Scenario: Copilot Enterprise in a Teams 1:1 internal call, started after ~ 15 minutes of talking.
Usages: Provided a bulleted list, free-form request wasn't as structured as the proposed prompt
Outcome:
6/10 for free form prompt
8/10 for suggested/curated prompt
Interaction Record
Replacements in the image are:
- Yellow - Customer name
- Blue - Internal persons name
- Light blue - name of person at customer
- Orange - "hallucination" customer / project names
Summarise what we talked about
Copilot
Here is a summary of what you talked
about:
- You discussed the pilot project for
copilot with <customer>, and the
tasks and templates you need to
prepare for it. - You leamed about the customer's
approach and hypothesis for using
copilot, and the different user
groups and personas they have. - You reviewed some of the data and
information that <internal name> has
gathered about the customer's
current state, productivity score,
and maturity model. - You planned to use a employee
feedback tool or forms to capture
the subjective feedback from the
users, and to create some power Bl
reports to measure the impact of
copilot. - You explored some of the features
and challenges of loop, and how to
use it for collaboration and note-
taking. - You also talked about some other
projects and activities you are
involved in, such as the invoke
workshop, the <no idea 1>, and the
<no idea 2> project.
Al-generated content may be incorrect
Summarize call so far
Copilot
Key Topis:
- Copilot pilot project with Jan
Farmer: <internal names> agreed to
work on some templates, survey
questions and power Bl reports to
measure the impact of copilot on
the users. - Hypothesis and user groups:
<internal names> planned to sync with
<customer contact>, the customer
contact, to understand their
hypothesis and user groups for the
pilot. - Loop tool: <internal names> explored
the features and limitations of the
loop tool for collaboration and
documentation.
Al-generated content may be incorrect
List action items
Suggest follow-up questions
