[HOLD for payment 2025-02-04] [HOLD for payment 2025-01-21][$250] Fix Proposal Police from commenting unexpected things on issues / Update to use JSON formatting

[thienlnam]

Example here: #54905 (comment)

The ProposalPolice bot commented something unexpected and this was due to the API call returning something unexpected which was not handled.

Solution:
Update the API to use structured responses which will ensure that the returned parameter is in a format we expect which should prevent situations like this in the future

Upwork Automation - Do Not Edit

• Upwork Job URL: https://www.upwork.com/jobs/~021877133911642803604
• Upwork Job ID: 1877133911642803604
• Last Price Increase: 2025-01-08
• Automatic offers:
• ikevin127 | Contributor | 105617348

Issue Owner

Current Issue Owner: @JmillsExpensify

[melvin-bot]

Job added to Upwork: https://www.upwork.com/jobs/~021877133911642803604

[melvin-bot]

Triggered auto assignment to Contributor-plus team member for initial proposal review - @ZhenjaHorbach (External)

[melvin-bot]

Triggered auto assignment to @JmillsExpensify (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details. Please add this bug to a GH project, as outlined in the SO.

[melvin-bot]

Upwork job price has been updated to $125

[melvin-bot]

📣 @ikevin127 🎉 An offer has been automatically sent to your Upwork account for the Contributor role 🎉 Thanks for contributing to the Expensify app!

Offer link
Upwork job
Please accept the offer and leave a comment on the Github issue letting us know when we can expect a PR to be ready for review 🧑‍💻
Keep in mind: Code of Conduct | Contributing 📖

[thienlnam]

Kevin is working on this - slack discussion here

[melvin-bot]

Upwork job price has been updated to $250

[ikevin127]

♻️ Plan

• update AI assistant's instructions to account for recently updated proposal template (new automated tests section)
• update AI assistant's instructions to enforce returning JSON structured data response (see slack discussion for details)
• upgrade and test new AI model for proposal-police to reduce cost of API usage (see slack discussion for details)
• update proposal-police GH action logic to match updated AI assistant's structured data response

[ikevin127]

Updates

• ✅ update AI assistant's instructions to account for recently updated proposal template (new automated tests section)
• ✅ update AI assistant's instructions to enforce returning JSON structured data response (see slack discussion for details): updated the instructions and took advantage of OpenAI's new Assistant Model configuration - Response format to request explicit json structured schema as response
• ✅ upgrade and test new AI model for proposal-police to reduce cost of API usage (see slack discussion for details): changed to gpt-4o-mini (cheapest - see slack discussion for details)

Here's the test repository to get an Idea of action response after the 3 changes mentioned above were implemented: ikevin127/expensify-proposal-testing#1 (comment)

♻️ Tomorrow will proceed with update proposal-police GH action logic to match updated AI assistant's structured data response and perform final testing before opening E/App PR.

[ikevin127]

♻️ Did some more testing and looks like the o1-mini model is not available with AI assistants. I started testing the different cases and neither gpt-4o-mini nor gpt-4o seem to follow the instructions when it comes to wrong proposal template case, even though it does return structured format, it doesn't seem to understand the context when it comes to the proposal template and the mandatory / optional sections.

The only one that's working as expected is the gpt-4-1106-preview model, which is what I use on my side for testing and what proposal police was based on from the beginning.

↪️ I will continue to test on my side with gpt-4-1106-preview, then I'll move forward with to the GH action code changes part. When I'm done I'll report on the testing and we can decide whether :expensify: wants to stay on the gpt-4-1106-preview model given the pricing or explore other alternatives.

After further discussions I'll do some prompt engineering to try and have it work on gpt-4o-mini or gpt-4o because otherwise the costs would be too high.

[ikevin127]

cc @thienlnam @marcochavezf PR is ready for review! 🚀

[ikevin127]

@thienlnam Will try a few test proposals here to see what's happening in the actions.

[thienlnam]

PR is merged, and since it's an action it is already taking effect. So considering 7 days from today as the payment date

[melvin-bot]

Reviewing label has been removed, please complete the "BugZero Checklist".

[melvin-bot]

The solution for this issue has been 🚀 deployed to production 🚀 in version 9.0.85-4 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

• Proposal-Police: GPT-4o Update with Structured Response #55108

If no regressions arise, payment will be issued on 2025-01-22. 🎊

For reference, here are some details about the assignees on this issue:

• @ikevin127 requires payment automatic offer (Contributor)

[melvin-bot]

@ikevin127 @JmillsExpensify @ikevin127 The PR fixing this issue has been merged! The following checklist (instructions) will need to be completed before the issue can be closed. Please copy/paste the BugZero Checklist from here into a new comment on this GH and complete it. If you have the K2 extension, you can simply click: [this button]

[ikevin127]

cc @JmillsExpensify for visibility

[JmillsExpensify]

Payment summary: $250 to @ikevin127 for PR.

[JmillsExpensify]

All paid out. I paid the difference from the earlier amount via a bonus in Upwork.

[melvin-bot]

Reviewing label has been removed, please complete the "BugZero Checklist".

[melvin-bot]

The solution for this issue has been 🚀 deployed to production 🚀 in version 9.0.89-8 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

• [No QA] Proposal Police™ Structured Response Follow-Up #55330

If no regressions arise, payment will be issued on 2025-02-04. 🎊

For reference, here are some details about the assignees on this issue:

• @ikevin127 requires payment automatic offer (Contributor)

[melvin-bot]

@ikevin127 @JmillsExpensify @ikevin127 The PR fixing this issue has been merged! The following checklist (instructions) will need to be completed before the issue can be closed. Please copy/paste the BugZero Checklist from here into a new comment on this GH and complete it. If you have the K2 extension, you can simply click: [this button]

[ikevin127]

❌ No need for additional payment here, as this was a follow-up PR.