Native Modules

[grabbou]

I've been thinking about the state of native modules and what we could do better to improve them. After working with few other React Native team members on the details, I think I am ready to share what I've came up with so far.

This issue is based on some feedback I gathered from #60. Since the scope is a bit broader than just linking libraries, I decided to open a new issue.

This document describes the current developer experience working with native modules, most important problems and areas where we could improve. The intent is for this issue to become umbrella for various RFCs that will be submitted as a part of it.

Glossary of terms

I want this issue to be super easy to understand, hence I decided to break down the terminology that I use throughout this issue. If you find anything unclear, please comment and I'll add it here.

• React Native library - a npm package that can contain one or many native modules
• Native Module - a module that extends capabilities of React Native. It usually ships with Android and iOS source code.
• Native Dependency - native module that depends on other Java and/or ObjC libraries, that may be distributed as source, or as compiled binaries.
• Linking a dependency - the process of adding files present in a native module to your iOS and Android projects
• react-native- React Native CLI
• react-native link - command to link a dependency to your project. It is a part of React Native CLI

Problem

Working with native modules in React Native can be overwhelming. From the library author perspective, it is not clear how native dependencies could be handled and what is the recommended structure for a module.

From the app developer perspective, it is not easy to consume React Native dependencies from the npm registry. Some of them are pure Javascript libraries that can be installed just like any other Javascript dependency. Others require additional steps to be performed (namely Android and iOS files to be added to respective iOS and Android projects). This can be confusing and increases the entry-barrier for the developers that want to extend basic capabilities of React Native with 3rd party modules.

See the reasons highlighted below for details.

Lack of standard

There is no standard or recommended way to create a React Native library. This problem is especially visible on iOS, where no built-in package manager for native dependencies is available (see "No support for native dependencies on iOS" section).

As a result, each React Native library tends to have a slightly different structure in order to satisfy their use-case. This makes it hard to provide a decent tooling support that can automate most of the repeated steps (see "Poor tooling support" section).

It is also not clear at a glance what steps need to be performed. This degrades the developer experience from a trivial “yarn add” to many complex instructions.

No support for native dependencies on iOS

Some React Native dependencies rely on native libraries in order to provide their functionality. It is a common practice to wrap a native SDK and expose it to React Native developers.

On Android, the open-source community settled on Gradle as the go-to package manager. However, iOS doesn’t have a built-in package manager that would handle downloading, compiling and linking native dependencies to a project. As a result, React Native developers are often required to perform a set of manual steps, some of them, requiring a decent native knowledge.

Few approaches to handling native dependencies have developed over the course of few years within the community, some of them are listed below:

• shifting the responsibility over to user-space. For example, react-native-fbsdk requires developers to install Facebook SDK themselves and make sure its already configured in their projects at the time of installing this library
• using a iOS tool specialised in handling native dependencies. The most popular one amongst the community and Expo seems to be CocoaPods. It relies on a Podfile (similar to package.json, where dependencies are defined). This is an equivalent of Gradle for iOS. React Native already provides CocoaPods Podspecs for its built-in modules
• checking in native files to repository. Similar to React Native shipping with precompiled Android sources for faster build times, some library developers opt to check-in SDKs and native dependencies to the Git repository. That way, they are distributed over through npm registry.

There's been a great discussion in #60 on how this could be addressed.

Poor tooling support

Since there is no recommended or the de facto standard way to create a React Native library, there are no commands that would support developers in making them. That contributes to a poor debugging experience (discussed in the next section).

There are few community driven initiatives, such as react-native-create-library (which is a perfect example of repeating format from a different module, in this case - react-native-share).

At the same time, on iOS, “react-native link” is unable to offer the level of reliability and performance that React Native developers would expect. Since there’s no standard way of managing React Native dependencies, it has to apply few heuristics in order to work (for example, locate Android and iOS source files using glob).

Debugging

The debugging experience while working with native modules could be improved. In order to test their libraries, developers need to create an example React Native app where they can require their library from source and make sure it meets their expectations.

This can be set up in many ways, each having a different developer experience.

For example, one of the most popular ways of setting up a demo project for the library is to run a “react-native init” command to create a stand-alone React Native app and make the library a dependency.

This is usually done using file protocol (symlinks are not supported) that tells a package manager to copy files from the specified location and move them over to node_modules, as if they were installed from the registry. You can see it live in react-native-fbads.

The above approach has the following drawbacks:

• developers need to install dependencies twice, once for the library itself (by running npm install or yarn install in the root of the project) and second time, for the demo app itself (this time, in the example/ folder)
• there are two versions of React Native installed that need to be kept in sync
  changes to the library are not respected in real-time - developers have to re-install dependencies for
• the demo app in order for the package manager to copy the new files again

Solution - Define the standard

We should describe how to create a React Native library and build reliable tooling around that. This should be done via series of RFCs.

Managing native dependencies

The standard should describe how to create a React Native library that has native code and native dependencies and how it can be exported for the end user in the easiest way possible.

While doing so, we should focus on maximising the strengths and capabilities of existing package managers for the given platform, such as Gradle and CocoaPods in order to reduce the overall maintenance cost and support of additional use-cases. Going this direction would make the specification really small in terms of the rules and make it familiar to those who have experience working with iOS and Android already.

One advantage is that Expo is also already using CocoaPods and Gradle to support "ejected" applications, which might be an additional driver for promoting this pattern throughout the community. That means we can share the effort to support the same set of tools for both Expo and React Native developers. All in all, Expo developers interact with React Native CLI once they eject from Expo.

Complementary tooling for scaffolding libraries

Creating packages for React Native’s been always pretty complex and manual process that can potentially lead to annoying issues down the road. It would be great to extend the capabilities of the “react-native” command line tools to support that use-case.

The suggested approach would be to provide support for:

• creating a native library that conforms to the standard,
• validating that a package is a valid library that conforms to a standard

See Flutter documentation for reference on how this can be addressed.

Better debugging experience

The standard project structure should provide great developer experience out of the box. That said, library generated with the react-native (see Improved Tooling section) should have an example app that avoids the drawbacks mentioned in the Debugging section.

A good example of such project structure is react-native-opentok that doesn’t copy library sources to example app’s node_modules, making the changes done to the library being reflected in the realtime.

Another one is react-native-camera that uses custom Metro configuration to require the library from source.

Whole-new linking experience

The purpose of the react-native link would be to automate adding a React Native library to a React Native project. It would only work for libraries that follow the standard library structure. That means, linking a library that has "Gradle" and "CocoaPods" inside.

Having a clear set of supported use-cases would make it much easier to make it reliable and reduce the overall cost of its maintanance. The current react-native link has been working in more-or-less its original form for around 2 years. That's a lot given the amount of updates that did happen to React Native.

The intent is for React Native developers to be able to run react-native install XYZ without worrying about the details of a particular library. With CocoaPods and Gradle, it wouldn't require much of the effort (especially compared with the current link maintanance) to offer such improvement.

Additional platforms

This proposal would be only targeted at Android and iOS (that are part of React Native right now). It would be great to work with Windows team on specifying the details for their platform. Right now, they can extend react-native link (and other CLI commands) on themselves. I would expect them to provide support for their standard using this mechanism too.

Next steps

I would appreciate your feedback on this document and encourage a discussion what we could do better in order to make this experience better for React Native developers.

The next step would be to submit an RFC with the standard structure for a library, how it should manage dependencies, how the example app should be provided for best debugging experience.

Rollout

Once the RFC gets merged, we will work inside react-native-cli to implement it. This will require some additional tools and commands to be created (they will be described in the RFC).

It is worth noting that React Native itself, as a framework, will remain neutral as far as the project structure and how the dependencies should be handled. The term "standard" is probably overused here. I decided to open this issue here, instead of react-native-cli repository for better visibility.

[rhdeck]

I’ve been working on these problems in my own tooling in iOS (as evinced by my react-native-swift-cli quickstart library) with the following propositions:

1. Deploy RN modules as Static Libraries.
2. Deploy external dependencies as CocoaPods
3. Define the external CocoaPod dependencies using react-native-pod
4. Use react-native-linknopod to replace react-native link so that RN modules that have podspecs don't get auto-imported to my podfile

This is a hodgepodge, but it solves my problem that podspecs otherwise "go rogue" when I try to make better devex with integrated pod support for dependencies.

I concur that standardizing on pods would be ideal. And that it would be a significantly breaking change. I don't think that's such a big cost - upgrading RN is a challenge every month or so today, so making a hard thing a little harder for more people - once - to get a smoother experience on the other side seems like a reasonable price.

I in the absence/in advance of a shift to all-pod all the time, I propose that one of the best things we could do is allow switching on react-native link to tell it to ignore podspecs. Would love to retire react-native-linknopod.

[zoontek]

A few things I would love to see 🙌🏼:

• The possibility to create a project using only CocoaPods. After, let's suppose react-native init ProjectName --cocoapods, a Podfile will be created with the exact same dependencies as the current project initialisation and nice comments for every subspecs.

Currently it's barely documented, and it's hard to understand which subspecs are necessary.

target 'ProjectName' do
  pod 'React', :path => '../node_modules/react-native', :subspecs => [
    'Core',
    'CxxBridge', # Include this for RN >= 0.47 -> include it in React/Core?
    'DevSupport', # Include this to enable In-App Devmenu if RN >= 0.43 -> same?

    'RCTAnimation', # Needed for FlatList and animations running on native UI thread
    'RCTNetwork',
    'RCTText',
    'RCTWebSocket', # Needed for debugging

    # Which subspecs can be added? Which are absolutely necessary? Which can be excluded?
    # https://github.com/facebook/react-native/blob/master/React.podspec
  ]

  pod 'yoga', :path => '../node_modules/react-native/ReactCommon/yoga' # include it in React/Core?

  # Why should I install this? I need context
  pod 'DoubleConversion', :podspec => '../node_modules/react-native/third-party-podspecs/DoubleConversion.podspec'
  pod 'glog', :podspec => '../node_modules/react-native/third-party-podspecs/glog.podspec'
  pod 'Folly', :podspec => '../node_modules/react-native/third-party-podspecs/Folly.podspec'

end

• A way to easily control libraries depencies in user land. For example react-native-maps relies on multiples variables. It could be nice to have a standardised way to handle it.

subDependencies = {
  // you can override sub dependencies versions
  "com.google.android.gms:play-services-base": "16.0.1",
  "com.google.android.gms:play-services-maps": "16.0.0",
  "com.google.maps.android:android-maps-utils": "0.5+",
}

And in libraries, something like:

dependencies {
  // dependency: (packageName: string, defaultVersion: string) => string
  compileOnly dependency("com.facebook.react:react-native", "+")
  implementation dependency("com.google.android.gms:play-services-base", "16.0.1")
  implementation dependency("com.google.android.gms:play-services-maps", "16.0.0")
  implementation dependency("com.google.maps.android:android-maps-utils", "0.5+")
}

Not sure if it's doable, no idea on how to tackle the same issue on iOS.

[grabbou]

Hey @zoontek, thank you for your comment!

The possibility to create a project using only CocoaPods.

The intent was to make it integrated with all projects by default. Adding it later would be complex, as you pointed out.

A way to easily control libraries depencies in user land

This is something @axe-fb was asking about (how do we deal with multiple libraries depending on a different version of another library).

@axe-fb, do you think react-native-maps example could be a good candidate to solving this problem?

[rhdeck]

Thinking about @zoontek point, maybe the app-level Podfile should be generated by RN - every build (where a build might be a single run of react-native link). The question of which dependencies to load in the podfile should be managed via package.json (a react-native-modules key?) I take a version of this approach in react-native-pod. Keeps RN orchestration in "nodeland" which is more broadly understandable (whereas podfiles and podspecs are a more specific knowledge that we don't want to require of RN devs). This also helps with some degree of dependency management by allowing RN-level conflict resolution (and sequencing, if that turns out to matter) to be managed at the podfile creation point.

[fson]

How do we deal with the additional dependency on Ruby which Cocoapods come with?

I think the best way to resolve this question is that CocoaPods also works with the default Ruby that comes pre-installed with macOS (and macOS is required to develop iOS apps with React Native), so you don't even need to install Ruby separately, if you don't want to.

[jdmunro]

Definitely would be great to see more guidance around how native modules should package up any third-party SDKs they depend on. I've seen all sorts, some where frameworks are bundled into the static library, but you typically need to embed them into the application anyway. Others where you manually provide the SDK yourself at the application level (which seems to work better IMO).

Better tooling for creating native modules would be a great win. react-native-create-library has been a good start but has some limitations: it would be great for a project like this to be adopted and become a hallmark for best practice.

[elicwhite]

Some thoughts on how we migrate the community and support legacy link:

I think the tool should continue supporting legacy linking with a flag like —useLegacyLink. If you try to link a package that hasn’t migrated to the new system without using the flag it should print a warning to the console saying that the package isn’t following the latest standard, asking the developer to open an issue on that project asking it to support the new system, and then telling them to rerun with —useLegacyLink.

[fkgozali]

This document describes the current developer experience working with native modules, most important problems and areas where we could improve.

I think it shouldn't be scoped to just native modules. This is about native dependencies (native modules, viewmanagers, future Fabric/TurboModule dependencies, etc).

We should describe how to create a React Native library and build reliable tooling around that.

I'd like to make sure that the standard includes specifications for how TurboModule/Fabric components should be built/packaged. A lot of this is still work in progress, but it's important to plan ahead for the future direction of native dependencies. A few things to consider:

• [React Fabric] Updating Build Configuration (CocoaPods, .xcodeproj, gradle) facebook/react-native#22989 may influence the requirements for Fabric components. I'd argue that it is a pre-req to figuring out the correct standard here.
• There will be a lot more C++ code for each module/component in Fabric/TurboModule. That means the standard needs to correctly prescribe the C++ build tooling requirement (e.g. version, type, etc)
• There will be codegen tool for Fabric components/TurboModule: CodeGen Discussion #92 -- how should lib author incorporate the codegen tooling?
• Related to prebuilt binary vs building from source, having C++ in play will mean more build conflict resolution to be done. Prescribing the strategy for this is going to be important. E.g. if a lib is built for C++ 17, but the app is stuck at C++14, there can be a mismatched in APIs/symbols.
• Note that there will be no more NativeModules/ViewManagers once Fabric/TurboModule is complete.
• We use BUCK heavily for fabric/TurboModule, and there have been discussions about using BUCK as source of truth (which could produce podspec/gradle etc as needed). That's a longer term discussions, but I think it should be covered in the standard if we believe in this direction.

Folks build C++ heavy integration may have opinions in this space. cc @matthargett, @acoates-ms

[axemclion]

Some thoughts on legacy link support

IMHO, we can still have a proposal without breaking the way existing modules are linked. Linking is only a pain when react-native modules depend on other native sdks. If that is not the case, always forcing gradle or cocoapods may result in the native module author having to publish 2 packages - one on npm, one of gradle or cocoapods locations.

[axemclion]

Some thoughts on upgrading

Another issue with the current linking story is that it modifies the XCode project or MainApplication.java during react-native link phase. As a result, all the information about the native module that is added, is now codified into the application.
If the structure of the app changes, most people have to blow away the android/iOS folders and reinstall dependencies. Instead, we could use an approach (like NS or Cordova) where react-native link (or install) simply add the information to a app.json, and the compilation of the projects actually generate the require files to be added to the project.

This may also be inline with how we could codegen for Fabric/TurboModules.

[jamonholmgren]

@axemclion I support that approach 100%.