Providing Feedback
Lucy Feedback
Lucy never leaves, never forgets, and becomes smarter every day through user interactions. Just like you would expect a new employee to grow professionally, Lucy is always improving. She learns through user feedback (reinforcement learning) and machine learning (unsupervised learning) which enables Lucy to provide the most relevant answers for every question.
Feedback, also referred to as user training, is instrumental to Lucy’s continued evolution with her customers. User feedback applied over time increases the relevance of the AI model underlying Lucy as well as makes the system more responsive to user inputs. In this article we will review the key topics relating to the feedback of Lucy.
Types of Feedback
Lucy’s learnings are primarily reinforced through four user generated actions:
- Answer Feedback a.k.a. Question and Answer (“QnA”) Pairs
- Verified Answers
- Lucy Assist
- Synopsis Feedback
Answer Feedback a.k.a. Question and Answer (QnA) Pairs
Upon asking a question Lucy will quickly present users with the 10 best answers she feels relevant based on the information she has available. Answers are prioritized by relevance and shaped by user feedback. All feedback, both positive and negative, will help her continually learn. Lucy learns through user interactions, with her primary feedback mechanism through the 👍 👎 available within every answer.
Figure 1 – Thumbs Up/Thumbs Down is Available on Every Answer.
After users have applied a thumbs up/thumbs down to specific answers within the results displayed, training data is stored in what is known as a Question-and-Answer Pair, or QnA Pair. The reason for this is this feedback is an association a user made between an answer and a question, and the system responds to that input appropriately. Thumbs up correlates to “This answer was helpful” and generates a positive trained QnA pair. Thumbs down correlates to “This answer was not helpful” and generates a negative trained QnA pair.
Verified Answers
Verified answers are a specific type of QnA pair with a different purpose. The QnA pairs detailed above allow users to tell Lucy feedback about an answer. Verified answers allow users to tell other users that this answer answers the question being asked. Verified answers have special user interface treatment and are ranked above all other answers in that question’s results.
Figure 2 – Verified Answer User Interface Treatment.
Lucy Assist
For each question, after returning results, Lucy asks the user “Did you find what you were looking for?” This question is intentionally broader than a QnA pair in that it tries to understand whether Lucy solved the problem for the user. Feedback to this area allows Lucy Administrators and support individuals to identify when content is not connected or when the system did not work as intended. Some Lucy Assist “No” responses are connected to automated ticketing systems, and some are connected to Question-and-Answer Forums like AnswerHub, Quora-style content which allow users to fuel new content into the system.
Figure 3 – Lucy Assist User Prompt and Flow.
Lucy Synopsis
Like QnA pairs, users may provide feedback via thumbs up/thumbs down 👍 👎 to the Lucy Synopsis created for a given question. This feedback allows Synopsis to get better over time by influencing the Synopsis, creating telemetry/reporting events, and generating product improvement opportunities.
Figure 4 – Thumbs Up/Thumbs Down is Available on Every Synopsis Response.
How QnA Pairs Get Applied
When a user creates a QnA pair, that information gets applied to Lucy in three ways: direct feedback, indirect feedback, and telemetry/reporting. Direct and indirect feedback are a pair and exist as part of Lucy’s Answer Engine to assist in retrieval of relevant content for future user queries. They both use QnA pairs, and they both make Lucy’s Answer Engine “smarter” at retrieving results. The final category is telemetry and reporting. Each of the three applications for trainings are covered in detail below.
Direct and Indirect Feedback
The direct feedback takes QnA pairs and applies it within Lucy’s backend architecture. Internally it is referred to as the Cache Collection. This allows users to see the results of their action quickly and provides an additional layer of boosting to the system. When users train an answer and then ask the question again right away and the ranking of results changed, that is the direct feedback system at work.
Indirect feedback goes into the AI Model and is internally referred to as the Ground Truth. This allows the AI to get smarter at matching key elements within the question and key elements within the answer to make the entire system smarter. If you train an answer and then ask the same question a week later and the ranking of results changed, that is the indirect feedback system at work.
One way to think about the comparison is direct feedback is more about the content, whereas indirect feedback is more about the context.
Telemetry/Reporting
The telemetry and reporting systems that receive training data allows Lucy Administrators and support Individuals to investigate, audit, triage, and report on training usage and performance system wide. Training data flows into the Activity Report and various internal reporting systems.