feat(SDK): add globs and dir support for artifact deps.

[destifo]

• Include glob and dir support for PythonRuntime deps.
• Include glob and dir support for DenoRuntime deps.
• add tests

The change includes support for declaring artifact dependencies through globs and dirs

MET-441

Migration notes

In the deps parameter for python.import(...) and deno.import(...), globs and dirs can be passed in addition to files.

• The change come with new or modified tests
• Hard-to-understand functions have explanatory comments
• End-user documentation is updated to reflect the change

Summary by CodeRabbit

• New Features

  • Introduced functionality for defining and deploying Typegraphs for Deno and Python runtimes.
  • Added support for defining a Deno runtime dependency graph with policies for test scenarios.

• Bug Fixes

  • Corrected the structure of the Deno.serve call in the serve function.

• Refactor

  • Enhanced method chaining for better readability in the MetaTest class.

• Tests

  • Updated test coverage reporting to include new Deno runtime test files.
  • Commented out and removed outdated test cases in Deno runtime tests.

• Chores

  • Updated platform specification in configuration files.

[destifo]

@CodiumAI-Agent /review

[CodiumAI-Agent]

PR Review 🔍

⏱️ Estimated effort to review [1-5]	4, due to the extensive changes across multiple files and languages, including TypeScript and Python, which involve complex features like glob pattern handling and directory management in dependency resolution. The PR also modifies core utility functions and introduces new test cases, requiring careful consideration of both functionality and potential side effects.
🧪 Relevant tests	No
⚡ Possible issues	Possible Bug: The use of is_glob function in fs_host.rs might not correctly handle edge cases where file names contain characters that resemble glob patterns but are not intended as such. This could lead to incorrect file handling.
	Performance Concern: The get_matching_files function in fs_host.rs could potentially lead to performance issues if the directory contains a large number of files or if the glob pattern is complex, as it iterates over all files and checks each one against the glob pattern.
🔒 Security concerns	No

Code feedback:

---

relevant file	typegraph/core/src/utils/fs_host.rs
suggestion	Consider optimizing the get_matching_files function by using more efficient glob matching libraries or algorithms that can handle large datasets more efficiently. This change is important to ensure that the function scales well with larger directories and more complex glob patterns. [important]
relevant line	pub fn get_matching_files(glob_pattern: &str) -> Result, String> {

---

relevant file	typegraph/core/src/utils/fs_host.rs
suggestion	Implement additional checks or a more robust pattern matching mechanism in the is_glob function to accurately determine if a string is a glob pattern. This will prevent potential bugs related to misinterpretation of file names as glob patterns. [important]
relevant line	pub fn is_glob(path: &str) -> bool {

---

relevant file	typegraph/core/src/utils/postprocess/deno_rt.rs
suggestion	Ensure that error handling is robust in resolve_globs_dirs to manage cases where glob patterns do not match any files or directories, preventing runtime panics or unhandled exceptions. [important]
relevant line	let resolved_deps = resolve_globs_dirs(deps)?;

---

relevant file	typegraph/core/src/utils/postprocess/python_rt.rs
suggestion	Similar to the Deno runtime, add comprehensive error handling around the use of resolve_globs_dirs in the Python runtime to gracefully handle errors and provide meaningful error messages to the user. [important]
relevant line	let resolved_deps = resolve_globs_dirs(deps)?;

---

[coderabbitai]

Walkthrough

The changes introduce new functionality and modifications across multiple files to support Deno and Python runtimes in the Typegraph framework. This includes adding new test files, modifying test coverage reporting, and updating dependency management. Additionally, significant updates have been made to utility functions and the core processing logic to handle glob patterns and directory resolutions for dependencies.

Changes

File Path	Change Summary
.ghjk/lock.json	Changed platform from "x86_64-linux" to "aarch64-darwin" and removed a specific entry.
.github/workflows/tests.yml	Added a new test file for Deno runtime coverage reporting.
typegate/tests/runtimes/deno/deno_dep.ts	Introduced functionality for defining a Deno runtime dependency graph for addition operations.
typegate/tests/runtimes/deno/deno_globs.py	Introduced functionality for defining and deploying a Typegraph for Deno runtime.
typegate/tests/runtimes/deno/deno_test.ts	Commented out and removed significant portions of code related to Deno runtime test cases.
typegate/tests/runtimes/python/python_globs.ts	Introduced functionality for defining and exposing Python functions using TypeGraph.
typegate/tests/utils/test.ts	Modified serve function, updated MetaTest class, and added logging for serialized data.
typegraph/core/Cargo.toml	Added glob = "0.3" dependency.
typegraph/core/src/utils/fs_host.rs	Added functions for handling glob patterns and directory resolutions.
typegraph/core/src/utils/postprocess/deno_rt.rs	Updated DenoProcessor to handle glob and directory resolutions for dependencies.
typegraph/core/src/utils/postprocess/python_rt.rs	Updated dependency resolution logic to use resolve_globs_dirs.

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

• X
• Mastodon
• Reddit
• LinkedIn

Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai generate interesting stats about this repository and render them as a table.
  • @coderabbitai show all the console.log statements in this repository.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to full the review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

Actionable comments posted: 1

Review Details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 5cc7729 and 0208e04.

Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

Files selected for processing (11)

• .ghjk/lock.json (3 hunks)
• .github/workflows/tests.yml (1 hunks)
• typegate/tests/runtimes/deno/deno_dep.ts (1 hunks)
• typegate/tests/runtimes/deno/deno_globs.py (1 hunks)
• typegate/tests/runtimes/deno/deno_test.ts (3 hunks)
• typegate/tests/runtimes/python_wasi/python_globs.ts (1 hunks)
• typegate/tests/utils/test.ts (4 hunks)
• typegraph/core/Cargo.toml (1 hunks)
• typegraph/core/src/utils/fs_host.rs (3 hunks)
• typegraph/core/src/utils/postprocess/deno_rt.rs (3 hunks)
• typegraph/core/src/utils/postprocess/python_rt.rs (2 hunks)

Files skipped from review due to trivial changes (3)

• .ghjk/lock.json
• .github/workflows/tests.yml
• typegraph/core/Cargo.toml

Additional comments not posted (8)

typegate/tests/runtimes/python_wasi/python_globs.ts (1)

12-32: Ensure the deps arrays in test_glob and test_dir correctly handle globs and directories as intended by the PR.

Verification successful

---

The verification process did not find the exact string 'deps: ["py/*.py", "py"]' in the file python_globs.ts. However, based on the provided code snippet, the deps arrays are used as follows:

• test_glob uses deps: ["py/*.py"] to specify a glob pattern.
• test_dir uses deps: ["py"] to specify a directory.

This matches the intended functionality as described in the review comment, where test_glob should handle a glob pattern and test_dir should handle a directory. Therefore, it appears that the deps arrays are correctly handling globs and directories as intended by the PR.

Scripts Executed

The following scripts were executed for the analysis:

---

Script:

#!/bin/bash
# Description: Verify that the `deps` arrays in `test_glob` and `test_dir` correctly handle globs and directories.

# Test: Search for the `deps` array usage. Expect: Correct handling of globs and directories.
rg --type python $'deps: \["py/\*\.py", "py"\]'

Length of output: 79

---

---

Script:

#!/bin/bash
# Description: Verify that the `deps` arrays in `test_glob` and `test_dir` correctly handle globs and directories in the TypeScript file.

# Test: Search for the `deps` array usage in the TypeScript file. Expect: Correct handling of globs and directories.
rg 'deps: \["py/\*\.py", "py"\]' typegate/tests/runtimes/python_wasi/python_globs.ts

Length of output: 86

typegate/tests/runtimes/deno/deno_globs.py (1)

18-39: Ensure the deps arrays in test_glob and test_dir correctly handle globs and directories as intended by the PR.

typegraph/core/src/utils/postprocess/python_rt.rs (1)

45-47: Ensure the resolve_globs_dirs function is used correctly to handle dependencies in the Python runtime.

typegraph/core/src/utils/postprocess/deno_rt.rs (1)

87-89: Ensure the resolve_globs_dirs function is used correctly to handle dependencies in the Deno runtime.

typegraph/core/src/utils/fs_host.rs (2)

238-254: Ensure the get_matching_files function correctly handles glob patterns as intended by the PR.

---

256-279: Ensure the resolve_globs_dirs function correctly resolves dependencies specified as globs or directories.

typegate/tests/runtimes/deno/deno_test.ts (1)

150-150: Ensure the new test for local imports is correctly implemented and integrates with the changes made in the Deno runtime.

typegate/tests/utils/test.ts (1)

191-193: Ensure the new method for deploying and testing typegraphs is correctly implemented and integrates with the changes made in the SDK.

[michael-0acf4]

Nice, can't wait for this to be merged!

[codecov]

Codecov Report

Attention: Patch coverage is 15.38462% with 33 lines in your changes are missing coverage. Please review.

Project coverage is 73.00%. Comparing base (8bf253b) to head (32278cf).

Files	Patch %	Lines
typegate/tests/utils/s3.ts	2.94%	33 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #698      +/-   ##
==========================================
- Coverage   73.14%   73.00%   -0.15%     
==========================================
  Files         117      117              
  Lines       14141    14171      +30     
  Branches     1415     1415              
==========================================
+ Hits        10343    10345       +2     
- Misses       3771     3799      +28     
  Partials       27       27              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

requested for review again and not working.