Skip to content

fix(sdk): node SDK cross-platform compatibility #1000

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Apr 16, 2025

Conversation

luckasRanarison
Copy link
Contributor

@luckasRanarison luckasRanarison commented Apr 15, 2025

  • FFI had to be used to make file descriptors blocking on unix systems because they are asynchrnous by default, meaning a file descriptor read immediately fails when there's no data instead of waiting. Deno abstracts this in its implementation but we have to handle edge cases by ourselves with NodeJS.
  • Fixes CI issues by upgrading ghjk to v0.2.2.

Migration notes


  • The change comes with new or modified tests
  • Hard-to-understand functions have explanatory comments
  • End-user documentation is updated to reflect the change

Summary by CodeRabbit

  • Bug Fixes
    • Improved compatibility for reading standard input on macOS and Linux systems, ensuring smoother and more reliable input handling.
  • Chores
    • Updated the version of the development environment tool used across workflows, documentation, and build configurations to v0.2.2.
  • Documentation
    • Updated a Python code generation example in the documentation for clearer reference.

Copy link

linear bot commented Apr 15, 2025

Copy link
Contributor

coderabbitai bot commented Apr 15, 2025

📝 Walkthrough

"""

Walkthrough

The changes introduce platform-specific handling for standard input file descriptor flags on macOS and Linux using the ffi-rs library. The code now ensures that the standard input is set to blocking mode by manipulating file descriptor flags with native fcntl system calls. Additionally, the logic for reading responses from standard input is simplified by directly using process.stdin.fd with fs.readSync, removing the previous manual management of /dev/stdin file descriptors. No changes were made to the declarations or signatures of exported or public entities.

The version string for the ghjk tool was updated from "v0.2.1" to "v0.2.2" across multiple files, including GitHub Actions workflows, Dockerfiles, configuration files, and documentation. These updates affect the version of the ghjk tool used in development, testing, release, and build environments without altering workflow logic or code behavior.

Changes

File(s) Change Summary
src/typegraph/specs/codegen/rpc/typescript/client.ts Added platform detection and dynamic import of ffi-rs for macOS/Linux. Set stdin to blocking mode using fcntl. Simplified input reading by using process.stdin.fd directly. Removed manual open/close of /dev/stdin. No changes to exported/public entity signatures.
.github/workflows/publish-website.yml, .github/workflows/release.yml, .github/workflows/tests.yml Updated environment variable GHJK_VERSION from "v0.2.1" to "v0.2.2". No other workflow changes.
CONTRIBUTING.md Updated ghjk tool version in installation instructions from "v0.2.1" to "v0.2.2".
tools/Dockerfile, tools/cross.Dockerfile Updated GHJK_VERSION build argument from v0.2.1 to v0.2.2, affecting the version of ghjk installed.
tools/consts.ts Updated constant GHJK_VERSION from "v0.2.1" to "v0.2.2".
whiz.yaml Updated environment variable GHJK_VERSION from "v0.2.1" to "v0.2.2" in setup section.
tools/deps.ts Updated all ghjk imports from version v0.2.1 to v0.2.2. Reformatted export statements and simplified function signature formatting without changing logic.
docs/metatype.dev/docs/reference/metagen/index.mdx Updated example Python file reference in code generation sample from remix_types.py to fdk.py.

Sequence Diagram(s)

sequenceDiagram
    participant App
    participant OS
    participant ffi-rs
    participant fs

    App->>OS: Detect platform (process.platform)
    alt Platform is macOS or Linux
        App->>ffi-rs: Dynamically import ffi-rs
        App->>ffi-rs: Open libc and define fcntl
        App->>ffi-rs: Get stdin flags, clear O_NONBLOCK, set new flags
        ffi-rs-->>App: Return result (error if non-zero)
    end
    App->>fs: Read from process.stdin.fd using fs.readSync
    fs-->>App: Return response data
Loading

Suggested reviewers

  • Yohe-Am
  • michael-0acf4
  • Natoandro
    """

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 6360018 and ec43c77.

⛔ Files ignored due to path filters (1)
  • deno.lock is excluded by !**/*.lock
📒 Files selected for processing (2)
  • docs/metatype.dev/docs/reference/metagen/index.mdx (1 hunks)
  • src/typegraph/specs/codegen/rpc/typescript/client.ts (3 hunks)
✅ Files skipped from review due to trivial changes (1)
  • docs/metatype.dev/docs/reference/metagen/index.mdx
🚧 Files skipped from review as they are similar to previous changes (1)
  • src/typegraph/specs/codegen/rpc/typescript/client.ts
⏰ Context from checks skipped due to timeout of 90000ms (6)
  • GitHub Check: bulid-docker (linux/amd64, custom-ubuntu-large)
  • GitHub Check: test-full
  • GitHub Check: lint-compat (macos-13, x86_64-apple-darwin, false)
  • GitHub Check: lint-compat (macos-14, aarch64-apple-darwin, false)
  • GitHub Check: test-website
  • GitHub Check: pre-commit

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@luckasRanarison luckasRanarison requested a review from a team April 15, 2025 11:15
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
src/typegraph/specs/codegen/rpc/typescript/client.ts (1)

93-93: Remove unnecessary fallback operator.
fs.readSync throws on error and always returns a number of bytes read, so the ?? 0 fallback is redundant.

-      bytesRead = fs.readSync(process.stdin.fd, buffer) ?? 0;
+      bytesRead = fs.readSync(process.stdin.fd, buffer);
📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between c0bdef5 and d21b1aa.

⛔ Files ignored due to path filters (1)
  • deno.lock is excluded by !**/*.lock
📒 Files selected for processing (1)
  • src/typegraph/specs/codegen/rpc/typescript/client.ts (3 hunks)
🔇 Additional comments (2)
src/typegraph/specs/codegen/rpc/typescript/client.ts (2)

5-5: Importing process module is appropriate.
No issues found; this ESM-safe import pattern is good practice in modern Node.js.


14-14: Destructuring the platform property is straightforward.
The usage is clear and consistent.

@luckasRanarison luckasRanarison force-pushed the met-876-fix-node-sdk-compatibility branch from b365726 to 230d33b Compare April 15, 2025 11:25
@luckasRanarison luckasRanarison force-pushed the met-876-fix-node-sdk-compatibility branch from 2f80aa3 to 6360018 Compare April 16, 2025 09:36
Copy link

codecov bot commented Apr 16, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.15%. Comparing base (c0bdef5) to head (ec43c77).
Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1000   +/-   ##
=======================================
  Coverage   81.15%   81.15%           
=======================================
  Files         143      143           
  Lines       18038    18041    +3     
  Branches     1967     1966    -1     
=======================================
+ Hits        14638    14641    +3     
  Misses       3382     3382           
  Partials       18       18           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@luckasRanarison luckasRanarison merged commit 408191c into main Apr 16, 2025
14 of 15 checks passed
@luckasRanarison luckasRanarison deleted the met-876-fix-node-sdk-compatibility branch April 16, 2025 13:23
@zifeo
Copy link
Member

zifeo commented Apr 17, 2025

@luckasRanarison does it change a configuration OS wide? your PR is awesome and may also be reflected in the com to explain this part of the config (next pr?)

@luckasRanarison
Copy link
Contributor Author

No, it's just scoped to the current process standard input, Deno does it behind the scenes. But I think it's not really needed to bring back the flag because that's our desired behavior.

},
});

const flags = fcntl([process.stdin.fd, F_GETFL, 0]);
Copy link
Contributor

@michael-0acf4 michael-0acf4 Apr 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth to mention that stdin/out can be shared by the parent/child processes. So, wouldn't changing the blocking mode also affect meta or something that runs meta? It might break some assumptions. The default is blocking after all.. even on linux or does the big guys like deno do the same?

Copy link
Contributor Author

@luckasRanarison luckasRanarison Apr 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, if the stdio is inherited this also affects the parent's io behavior but it's completely safe because typegraph script processes are spawned with piped stdio. Also just a correction, it's async by default...

Copy link
Contributor

@michael-0acf4 michael-0acf4 Apr 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

piped stdio

Got it.

it's async by default...

Ubuntu 24.04
image

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ubuntu 24.04
image

Yes

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wait. non block flag 0 is async?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

const result = fcntl([process.stdin.fd, F_SETFL, flags & ~O_NONBLOCK]);

would no op on linux (if we ignore syscalls)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same thing on mac, so wouldn't it just no op in all cases?

mac

Copy link
Contributor

@michael-0acf4 michael-0acf4 Apr 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@luckasRanarison ah, I think I get it.

  1. It no ops on the blocking flag (if already blocking).
    2. But it clears the over flags => so that made it work?

Copy link
Contributor

@michael-0acf4 michael-0acf4 Apr 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the spam but I think I managed to narrow down the behavior out of curiosity.

For future references, it is platform + runtime specific only behavior (this is why I got no ops in all cases on Python3 even on MAC like the screenshots above).

  1. flag is 0 on (Node, Linux) and (Deno, Linux), this is why it worked without any changes.
  2. flag is 0 on (Deno, MAC) too, this is why it worked without any changes (and also (Python, MAC)).
  3. flag is 4 on (Node, MAC), and as such it does not no op but do reset it but...

.. weirdly enough.. even though flag is 4 on (Node, MAC), uncommenting the clear op const result = fcntl([process.stdin.fd, F_SETFL, flags /*& ~O_NONBLOCK*/]) which should be no op (so should EAGAIN) makes it work and removing this line makes it EAGAIN. That suggests either a bug on Node side or that what that was retrieved was actually not the actual current default state on MAC.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants