AI Assistant: Get feedback on AI responses

[christad92]

Here are a few things we want to do:

• Add a flag that lets user report an unsatisfactory response with comment.
• A posthog survey to collect experience.

[josephjclark]

This needs more design detail. But this is required before general release (especially if we're not going to post chat logs to slack yet)

[christad92]

The ideal solution here for me is to allow users to flag a response from the assistant. We should have a text under the responses that say, flag this response and on click, it should open a modal that asks the user to select the reasons(s) for flagging the response- incorrect information, incomplete answer, irrelevant answer, confusing or unclear, too long or verbose answer, too short or lacking detail, outdated information, not based on documentation, formatting issues, and other.

This "flag this answer" feature should only show when a user hovers over the answer from the assistant.

When flagging a response, users can select more than one reason and should also provide more details about their reason for flagging the response. The responses will be saved in the db (maybe in a new table).

[christad92]

@josephjclark what do you think about this?

[elias-ba]

Here is how Claude does it (same as your proposal @christad92), and I think it's really cool.

Click on these

For positive feedback

For negative feedback

[josephjclark]

@christad92 to investigate whether we can attach the AI-chat session ID to a posthog survey.

Then we can have a really simple flow like:

• User clicks "Give feedback on / report this answer"
• We pop up a posthog survey
• We attach the session id to posthog so we can track it
• we ask a few questions
• job done

[theroinaochieng]

@elias-ba thanks for picking this up! This will unblock @christad92 on setting up the PostHog survey. Here's the proposed implementation on the front end:

The "Give us feedback on your answers" should:

1. be coloured like the other hyperlinks at the bottom
2. trigger the PostHog survey as shown in the comments above (this is set up on the PostHog dashboard)
3. provide Ayo with a trigger that he can link on the PostHog dashboard to prompt the survey

[christad92]

@elias-ba @theroinaochieng this is very good work! I have tested this on staging with PostHog but I have feedback. See below:

Can we change this message to "Flag this answer"?

[elias-ba]

@christad92 @theroinaochieng https://github.com/OpenFn/thunderbolt/pull/427