feat: add Microsoft Prompty file format support #4840

mldangelo · 2025-07-09T21:58:59Z

Summary

This PR adds comprehensive support for Microsoft's Prompty file format (.prompty files), enabling users to define prompts with metadata, model configuration, and content in a single file.

Features

Core Functionality

YAML frontmatter parsing for metadata and configuration
Chat and completion API support
Nunjucks templating (Jinja2-compatible) for variable substitution
Environment variable support with ${env:VAR_NAME} syntax
Sample data that serves as default variable values
Multi-turn conversation support
Image support using markdown syntax
Role markers including shorthand (e.g., "A:" for assistant)

Provider Configuration Mapping

Azure OpenAI: Maps endpoint, deployment, API version, and API key
OpenAI: Maps model name, organization, and API key
Azure Serverless: Maps endpoint configuration

Examples Included

Basic examples: customer service, technical support, completion API, environment variables
Advanced examples: Azure configuration, image analysis
All examples tested and working with the echo provider

Documentation

Comprehensive section added to site/docs/configuration/prompts.md
Example READMEs with clear usage instructions
All user-facing documentation in the appropriate locations

Testing

17 unit tests covering all functionality
Integration tests for file loading and glob patterns
All tests passing

Implementation Notes

Uses Nunjucks which is largely compatible with Jinja2 (Microsoft's spec)
Environment variables are resolved at runtime using Node.js process.env
Sample data is implemented as a prompt function that merges with test variables
Follows the existing promptfoo architecture for file processors

Breaking Changes

None. This is a new feature that doesn't affect existing functionality.

Checklist

- Add new processor for parsing and handling .prompty files - Support YAML frontmatter with model configuration and metadata - Convert role-based content (system, user, assistant) to chat format - Support both chat and completion API types - Map Azure OpenAI, OpenAI, and other model configurations - Implement sample data functionality for default variable values - Add comprehensive test coverage (14 unit tests, 3 integration tests) - Update documentation with detailed Prompty section - Add example prompty files demonstrating various features - Support for images in prompts using markdown syntax - Template rendering using existing Nunjucks engine This allows users to seamlessly use their existing Microsoft Prompty files with promptfoo's evaluation and testing capabilities.

- Add support for multiple template engines in prompty files - Default remains Jinja2/Nunjucks for Microsoft compatibility - Optional Handlebars support via 'template: handlebars' in frontmatter - Register common Handlebars helpers (eq, ne, lt, gt, lte, gte) - Add comprehensive tests for both template engines - Update documentation with template engine examples - Create advanced examples demonstrating both syntaxes - Add handlebars dependency to package.json This allows users to choose their preferred template syntax while maintaining backward compatibility with the Microsoft Prompty spec.

- Document that Microsoft Prompty uses Jinja2, not Handlebars - Identify missing core features like environment variables - Recommend removing Handlebars support in favor of spec compliance - Suggest focusing on core features users actually need

- Remove Handlebars support to comply with Microsoft spec - Add environment variable support using ${env:VAR_NAME} syntax - Update input/output type definitions in interface - Simplify documentation to focus on Nunjucks (Jinja2-compatible) - Remove Handlebars examples and dependencies - Add environment variable example - Update all tests to reflect changes BREAKING CHANGE: Removed Handlebars template engine support. Prompty files now only support Nunjucks (Jinja2-compatible) templating as specified by Microsoft.

Add comprehensive support for .prompty files following Microsoft's specification: Features: - YAML frontmatter parsing for metadata and configuration - Chat and completion API support - Nunjucks templating (Jinja2-compatible) for variable substitution - Environment variable support with ${env:VAR_NAME} syntax - Sample data that serves as default values - Multi-turn conversation support - Image support using markdown syntax - Role markers including shorthand (A: for assistant) Configuration mapping: - Azure OpenAI: Maps endpoint, deployment, API version - OpenAI: Maps model name and organization - Azure Serverless: Maps endpoint configuration Examples included: - Basic examples: customer service, technical support, completion - Advanced examples: Azure configuration, image analysis - Environment variable usage demonstration Documentation: - Added comprehensive section to prompts.md - Implementation notes in docs/prompty/ - Example READMEs with usage instructions Testing: - 17 unit tests covering all functionality - Integration tests for file loading - Examples verified with echo provider

Keep documentation in the main site/docs folder only

use-tusk · 2025-07-09T21:59:09Z

⏩ No test execution environment matched (fb6bdd8) View output ↗

View output in GitHub ↗

View check history

Commit	Status	Output	Created (UTC)
`873acdd`	⏩ No test execution environment matched	Output	Jul 9, 2025 9:59PM
`b7df2df`	⏩ No test execution environment matched	Output	Jul 9, 2025 11:42PM
`fb6bdd8`	⏩ No test execution environment matched	Output	Jul 9, 2025 11:47PM

gru-agent · 2025-07-09T21:59:11Z

TestGru Assignment

Summary

Link	CommitId	Status	Reason
Detail	`873acdd`	✅ Finished

History Assignment

Files

File	Pull Request
src/prompts/processors/prompty.ts	🚫 Skipped (There's no need to update the test code)
src/prompts/index.ts	🔴 Closed #4845

Tip

You can @gru-agent and leave your feedback. TestGru will make adjustments based on your input

coderabbitai · 2025-07-09T22:04:48Z

📝 Walkthrough

Walkthrough

This change introduces comprehensive support for the Microsoft Prompty (.prompty) file format throughout the codebase. It adds a new processor for .prompty files, updates constants and file path utilities to recognize the extension, and integrates .prompty file handling into the prompt processing workflow. New documentation is provided, including detailed guides and examples for both basic and advanced Prompty usage. The change also adds new dependencies and extensive tests covering parsing, configuration mapping, environment variable resolution, and edge cases for .prompty files. Example Prompty files and configuration YAMLs are included to demonstrate usage.

Suggested reviewers

ladyofcode
typpo

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 ESLint

If the error stems from missing dependencies, add them to the package.json file. For unrecoverable errors (e.g., due to private dependencies), disable the tool in the CodeRabbit configuration.

npm error Exit handler never called!
npm error This is an error with npm itself. Please report this error at:
npm error https://github.com/npm/cli/issues
npm error A complete log of this run can be found in: /.npm/_logs/2025-07-09T22_00_52_901Z-debug-0.log

✨ Finishing Touches

📝 Generate Docstrings

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

‼️ IMPORTANT
Auto-reply has been disabled for this repository in the CodeRabbit settings. The CodeRabbit bot will not respond to your replies unless it is explicitly tagged.

Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai explain this code block.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and explain its main purpose.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai generate docstrings to generate docstrings for this PR.
@coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

coderabbitai

Actionable comments posted: 10

🔭 Outside diff range comments (1)

examples/prompty-advanced/promptfooconfig.yaml (1)
1-20: Fix configuration file structure to follow coding guidelines.

This configuration file is missing required elements and doesn't follow the specified field order:

Missing YAML schema reference at the top

Missing description field

Wrong field order (should be: description, env (optional), prompts, providers, defaultTest (optional), scenarios (optional), tests)

Apply this diff to fix the structure:
+# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
+description: Advanced Prompty examples demonstrating Azure configurations, multi-turn conversations, and image analysis
+
 prompts:
   - azure_chatbot.prompty
   - image_analysis.prompty

🧹 Nitpick comments (3)

examples/prompty-advanced/README.md (1)
31-37: Add language specifications to code blocks.

The static analysis correctly identifies that fenced code blocks should have language specifications for better syntax highlighting and accessibility.

Add language specifications to the code blocks:
-```
+```jinja
 {% if topic == "quantum computing" %}
 Quantum computing explanation...
 {% else %}
 General explanation...
 {% endif %}
-```
+```

-```
+```jinja
 {% for item in items %}
 - {{item.name}}: {{item.description}}
 {% endfor %}
-```
+```

-```
+```jinja
 Hello {{name | upper}}!
 The date is {{date | date("YYYY-MM-DD")}}.
-```
+```


Also applies to: 41-45, 49-53

</blockquote></details>
<details>
<summary>test/prompts/processors/prompty.test.ts (1)</summary><blockquote>

`345-347`: **Consider using undefined assignment for better performance**

While acceptable in test code, using `delete` can impact performance.

```diff
     // Clean up
-    delete process.env.TEST_API_KEY;
-    delete process.env.TEST_ENDPOINT;
+    process.env.TEST_API_KEY = undefined;
+    process.env.TEST_ENDPOINT = undefined;
src/prompts/processors/prompty.ts (1)
118-118: Consider handling edge cases in image URL regex

The current regex !/\[(.*?)\]$(.*?)$/g might have issues with URLs containing parentheses, which are valid in URLs.

Consider using a more robust regex or a markdown parsing library for better edge case handling:
-      const imageRegex = /!\[(.*?)\]$(.*?)$/g;
+      // Handle URLs with parentheses by matching balanced or escaped parens
+      const imageRegex = /!\[(.*?)\]$(<[^>]+>|[^\s)]+(?:\([^\s)]+$[^\s)]*)?)\)/g;
Alternatively, consider using a proper markdown parser for more reliable image extraction.

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 5f72d38 and 873acdd.

⛔ Files ignored due to path filters (1)

package-lock.json is excluded by !**/package-lock.json

📒 Files selected for processing (20)

.gitignore (1 hunks)
examples/prompty-advanced/README.md (1 hunks)
examples/prompty-advanced/azure_chatbot.prompty (1 hunks)
examples/prompty-advanced/image_analysis.prompty (1 hunks)
examples/prompty-advanced/promptfooconfig.yaml (1 hunks)
examples/prompty-basic/README.md (1 hunks)
examples/prompty-basic/completion_example.prompty (1 hunks)
examples/prompty-basic/customer_service.prompty (1 hunks)
examples/prompty-basic/env_example.prompty (1 hunks)
examples/prompty-basic/promptfooconfig.yaml (1 hunks)
examples/prompty-basic/technical_support.prompty (1 hunks)
package.json (1 hunks)
site/docs/configuration/prompts.md (1 hunks)
src/prompts/constants.ts (1 hunks)
src/prompts/index.ts (2 hunks)
src/prompts/processors/prompty.ts (1 hunks)
test/prompts/constants.test.ts (1 hunks)
test/prompts/index.test.ts (4 hunks)
test/prompts/processors/prompty.test.ts (1 hunks)
test/prompts/utils.test.ts (1 hunks)

🧰 Additional context used

📓 Path-based instructions (10)

`**/*.{ts,tsx}`: Use TypeScript with strict type checking.

**/*.{ts,tsx}: Use TypeScript with strict type checking.

📄 Source: CodeRabbit Inference Engine (CLAUDE.md)