[Due for payment 2025-05-08] Store prebuilt versions of our custom React Native artifacts

[roryabraham]

Coming from: https://expensify.slack.com/archives/C05LX9D6E07/p1739874783843659

Proposal: Reduce Build Time by Optimizing React Native Compilation

Background

Our build times for standalone NewDot and HybridApp have a significant impact on developer productivity and CI efficiency. Faster builds allow for quicker iterations, reducing the feedback loop when testing new changes. Currently, our build times remain high despite optimizations, affecting both local development and CI runs.

Problem

When engineers push new changes, the long build times delay feedback cycles, which slows down development.

Solution

We will store build artifacts for our patched React Native versions instead of building React Native from source each time. These stored artifacts will be regenerated only when patches on main or the React Native version change, which happens rarely. This will involve:

• Adding a system to store and retrieve prebuilt artifacts (e.g. maven repository).
• Automating version tracking to detect patch changes.

[mateuuszzzzz]

Hi! I'd like to work on this issue 😅🚀

[roryabraham]

Please provide a proposal for a versioning schema that's compliant with https://docs.oracle.com/middleware/1212/core/MAVEN/maven_version.htm#MAVEN8855

I think that we'd want to basically use the RN base version, for example 0.76.3 or 0.77.0-rc1, and then tack on a hash of all our react-native patches. But I'm not familiar enough with the rules of maven build numbers to know how this would work.

[mateuuszzzzz]

Hi! We're still exploring our options here. Nevertheless, I find this resource very useful for further investigation.

Ideally, we would like to maintain the following version scheme: {rn-version}-{our version for current rn-version starting from digit 1}.

...and then tack on a hash of all our react-native patches

When appending a hash to the version, we might encounter issues related to Maven's version ordering. Since the hash would be compared lexicographically, we could lose information about which hash corresponds to a newer version. To resolve this, we can store the hash in the build artifact's metadata and maintain a mapping of

{hash value for patch} -> {proper Maven version} somewhere.

[mateuuszzzzz]

Hi! I wanted to share an update.

Recently, I investigated how artifacts are created in the react-native repository. I managed to create a script that replicates this process locally and publishes the artifacts to a local Maven repository, allowing my local React Native apps to use them.

I have started working on a workflow that will publish these patched artifacts to a public Maven repository. There are two challenges to address:

1. we need our unique publishingGroup to avoid conflitcs with com.facebook.react
2. we need previously mentioned versioning scheme

First issue can be resolved by using react.internal.publishingGroup
e.g. react.internal.publishingGroup=com.expensify.react in gradle.properties
This property is successfuly used by this react-native fork: react-native-tvos

They also define the same versioning scheme that I suggested here. So I think It's totally fine to follow this idea.

Here's the quote from README:

Releases of react-native-tvos will be based on a public release of react-native; e.g. the 0.76.0-0 release of this package will be derived from the 0.76.0 release of react-native. All releases of this repo will follow the 0.xx.x-y format, where x digits are from a specific RN core release, and y represents the additional versioning from this repo.

[roryabraham]

sounds like we're close then? Let me know when you're ready for us to create a public maven repo and I can create a task for someone to make it

[mateuuszzzzz]

sounds like we're close then? Let me know when you're ready for us to create a public maven repo and I can create a task for someone to make it

I think we're getting closer. We just need to bring all parts together, but overall, the separate pieces are ready. We could discuss details regarding maven repository. Is there a preferred method for hosting it? E.g. we could use MavenCentral or self-hosted Nexus Repository Manager. We could also try to host plain maven repository on S3.

[roryabraham]

Not sure I'm too passionate either way. If MavenCentral is an option, let's just use that?

[mateuuszzzzz]

Sure! The only drawback of Maven Central is that the process of adding our repository there is quite complex, and we have less control. For example, we cannot delete a release version of a package, we can only delete SNAPSHOT versions, so we need to be cautious with the publication process.

One interesting alternative I found today is GitHub Packages. It's very easy to publish, and we can delete it anytime if anything goes wrong. Unfortunately there is one drawback. Downloading packages (even public ones) requires authentication with a GitHub token. So, users locally would need to have the gh CLI installed and a token with the necessary scopes.

Then, we would need to access it, for example, like this:

GH_TOKEN=$(gh auth token) 
GH_USERNAME=$(gh api user | jq -r '.login')

and read those credentials in build.gradle:

repositories {
    maven {
        url "https://maven.pkg.github.com/org-name/repo-name"
        credentials {
            username = System.getenv("GH_USERNAME")
            password = System.getenv("GH_TOKEN")
        }
    }
}

I'm leaving the final decision to you 😅

[mateuuszzzzz]

Regarding other updates:

I'm integrating a versioning mechanism with GitHub workflow. This is the last missing piece, and once it's ready, I will share a demo of the POC here. Then we can start adapting it in the App repository.

[roryabraham]

Looking forward to seeing the POC here. GitHub Packages seems like a great option to me, since we use the platform so heavily, it's great to not add another 3rd party system. 👍🏼

[mateuuszzzzz]

Looking forward to seeing the POC here. GitHub Packages seems like a great option to me, since we use the platform so heavily, it's great to not add another 3rd party system. 👍🏼

Awesome! Sorry for the delay, I’ve been handling a lot of high-priority crashes recently, but I’ve started moving the POC to the App repository.

Nevertheless, If you'd like to see current version of POC on new RN project here's a repository I was working on https://github.com/mateuuszzzzz/rn-android-artifacts

Here is workflow that handles publication process https://github.com/mateuuszzzzz/rn-android-artifacts/blob/main/.github/workflows/buildAndPublishArtifacts.yml

And example package published by mentioned workflow: https://github.com/mateuuszzzzz/rn-android-artifacts/packages/2440639

As I'm moving this workflow to App repository I need to refactor it a bit, including:

1. Usage of immutable hashes and our composite actions in place of direct usage (e.g. setup node)
2. I also store patch_hash->version mappings in repository (mappings.json) file. I think we can eliminate this additional file since we store the patch hash in the pom.xml file for each release, allowing us to query the correct version.

[roryabraham]

I see your POC workflow uses workflow_dispatch, should we instead run it whenever code is pushed to main that changes react-native either in the package.json or via a patch?

[mateuuszzzzz]

Yeah, this is final strategy I'd like to implement. workflow_dispatch was mainly useful for debugging purposes😅

[mateuuszzzzz]

I created a draft PR where I'll be moving the entire POC logic. I also simplified the publication process a bit, so now we can publish directly from the App repository. I'll post a TODO list in the PR's description.

[roryabraham]

Just left a big review on the PR. Overall it looks very promising

[melvin-bot]

Reviewing label has been removed, please complete the "BugZero Checklist".

[melvin-bot]

The solution for this issue has been 🚀 deployed to production 🚀 in version 9.1.38-4 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

• [No QA] Fix artifact workflow to use negated git diff exit status #60958

If no regressions arise, payment will be issued on 2025-05-08. 🎊

For reference, here are some details about the assignees on this issue:

• @mateuuszzzzz does not require payment (Contractor)

[melvin-bot]

Skipping the payment summary for this issue since all the assignees are employees or vendors. If this is incorrect, please manually add the payment summary SO.

[mateuuszzzzz]

I created a draft PR with 2 plugins. One is used during configuration and the other one during build phase. I'm working on another HybridApp PR that will use those plugins in Mobile-Expensify repository.

Here's what I'd like to address now:

1. New Gradle plugins need more descriptive errors and logs to guide users through the process. Currently, it's not obvious what's happening behind the scenes. Also, I think it's worth adding a README with instructions on how to use the prebuilt artifacts locally and how to disable them if needed.
2. I’ve identified the root cause of the issues with HybridApp artifacts, so I’ll prepare a follow‑up PR with a fix shortly.

Also, I’m leaning back toward our previous approach of identifying artifacts by contents hash. I’ve found it difficult to reliably verify whether a given commit matches an artifact’s version. There are numerous edge cases, such as staged versus unstaged changes. That need to be accounted for all at once, which complicates the implementation.

[mateuuszzzzz]

Hi, I wanted to share an update.

I prepared a few PRs to bring prebuilt artifacts to main so I'll describe next steps here.

1. This follow-up PR is ready and was already reviewed. Feel free to take a look @roryabraham

2. I prepared a follow-up PR that reintroduces patch change detection based on content hash. I'm reaching my EOD today and need to test it thoroughly on my fork tomorrow before we merge it. I'll let you know once it's ready.

3. There are two PRs ready for initial review

• NewDot PR: [No QA] Consume prebuilt react-native artifacts #61703,
• OldDot PR: https://github.com/Expensify/Mobile-Expensify/pull/13563

NewDot PR is currently missing implementation of hasDiff function and instruction in README. I'll add it once we merge #61862.

[mateuuszzzzz]

@roryabraham all PRs are ready. In case you'd like to start reviewing them, this is the correct order of reviewing/merging:

1. [No QA] Fix Conditional Logic in publish.gradle patch and publishReactNativeAndroidArtifacts.yml #61779
2. Update artifacts workflow to verify diff in patches based on hash of their contents #61862
3. [No QA] Consume prebuilt react-native artifacts #61703
4. https://github.com/Expensify/Mobile-Expensify/pull/13563

[mateuuszzzzz]

Also, after merging #61862 all existing patches will become outdated. Before merging this PR, we can delete existing packages to have a "clean start". However, this is not a blocker.

[mateuuszzzzz]

I noticed one more edge case that we should handle in order to make the workflow more resilient to errors. If we merge more than one PR that modifies react-native patches in a short time window, the workflow that finishes last will fail due to a conflicting Maven package version. Initially, they will read the same latest version from GitHub Packages and will try to publish the same version twice.

I'm curious if we already have solutions for this type of problem in our existing workflows. If not, I will try to come up with a solution that resolves the mentioned issue. Currently, I'm leaning towards enabling concurrency rules per matrix job. cc: @roryabraham

[roryabraham]

Yeah, concurrency is indeed the go-to solution for such a problem. I would think that a workflow-level concurrency key would be sufficient

[mateuuszzzzz]

Today I tested the workflow and discovered that if we merge a PR modifying patches first in the App repository and then another in the Mobile-Expensify repository, the first workflow that builds artifacts for both apps gets canceled, and only the second workflow that builds artifacts for HybridApp runs. As a result, the artifacts for the standalone NewDot build are missing. Mentioned issue occurs at both the workflow and job levels of concurrency.

I suggest refactoring the workflow by splitting the current jobs into two separate ones: the first job would check whether new artifacts need to be built, and the second job would handle building and publishing them. This way, we could apply concurrency control to the second job.

[roryabraham]

That sounds reasonable. What concurrency group key will you use for the publish job?

[mateuuszzzzz]

That sounds reasonable. What concurrency group key will you use for the publish job?

I wanted to use ${{ github.workflow }}-${{ github.job }}-${{ matrix.is_hybrid }}. I created a draft PR where I'm testing these changes to validate the overall workflow and concurrency setup.

EDIT: Marked as ready for a review

[mateuuszzzzz]

@roryabraham I removed HOLD from plugin PR, this one is already adjusted to recently merged PR

[mateuuszzzzz]

By the way, once those changes are merged, we can start gradually enabling prebuilt artifacts. First, we can enable them for local builds by default (this will require setting patchedArtifacts.forceBuildFromSource=false in both gradle.properties files). Once we're confident that it works on developers' machines, we can proceed with enabling it for CI builds.