install and link native modules from npm (as alternative to cocoapods/carthage)

[mauron85]

Introduction

Following is mostly valid for iOS.
Currently react-native-cli doesn't provide any support for plugin developers to install and link native modules like frameworks, static libraries, projects.

One can write custom link script and register it in rnpm's postlink phase and "misuse" not very well documented local-cli link scripts. However even with this approach, plugin needs to carry it's native dependencies. With more and more packages, this is slowly becoming a problem. I'm already aware of clang's linker error "IOS build error duplicate symbols for architecture x86_64" caused by two plugins linking same library.

Another popular option is to manage plugin dependencies via cocoapods.
However IMO cocapods/carthage are kind of alien in world of react-native. User need to know, how to install it, how to create podfile, etc. Linking modules should be implementation detail and users should be able to link all modules with just single react-native link command.

The Core of It

For all above reasons I decided to work on project I internally call native-modules. It will:

• augment common native libraries (like fmdb) with package.json with instructions how it should be linked in react-native project. Such a modules will be published to npm and could be used by any rn plugin.
• provide new react-native linker (and new clean xcode project parser) which will install and link all dependencies registered in app package.json

Discussion points

I understand I'm pretty brief here, but before going to deep into describing specification I would like know, if there is broad interest in such a thing.

[kelset]

cc @empyrical @grabbou

[matt-oakes]

This is something which has been on my mind recently as well. The experience as it stands at the minute only partly automates the process and leaves a lot of room for errors, especially with users who are not experienced with the native build systems.

Android is a fairly solved issue with the Gradle dependency system and most libraries making use of the rootProject.googlePlayServicesVersion convention.

iOS, however, needs some work.

For the Apple provided iOS frameworks, we could simply allow the React Native libraries to specify which ones they require in a standard way (for example react-native-location needs CoreLocation and react-native-device-info requires SystemConfiguration). The react-native link command can then ensure the correct ones are included in the targets linked libraries.

For third-party libraries, I think there are two possible directions this could be taken:

Create a native-module like system

As you mentioned, we could create a system where the required libraries are built and distributed over the npm registry. This would allow the libraries to specify their dependency on them and react-native link to correctly link them.

Advantages

• No need to learn about or use another dependency manager.

Disadvantages

• Requires each native library to be packaged, built, and distributed on NPM to be used. This would be fine for common libraries but would discourage use of smaller libraries.

Make use of an existing dependency manager

We could also leverage an existing iOS dependency manager in the same way that we do with Gradle for the Android apps. Cocoapods or Carthage would be the most obvious choices. RN libraries should be able to specify that they require a third-party library in a standard way (in a custom section of the package.json maybe?) and then the react-native link system should be able to send that information to the dependency manager to download and link the dependencies.

Advantages

• Most libraries for iOS are already packaged for one or both of these systems so any iOS library can easily be used.

Disadvantages

• The user might need to be aware of how the dependency system works, in the same way which they do right now with Gradle on Android.

---

Personally, my vote would be to build on top of Carthage. This avoids the need to reinvent an entire dependency manager for iOS, but it is also a lot simpler than Cocoapods. It can work by specifying the dependencies in a Cartfile (could be generated from the dependency requirements of the RN libraries). It then builds the *.a libraries for each dependency and then react-native link could automatically ensure that these files are linked in the XCode target.

To me, this strikes a good balance of being each for users and easy for RN library and iOS library developers.

---

Edit: Looking at how NatveScript plugin are specified they seem to have it set up very nicely (https://docs.nativescript.org/plugins/plugin-reference). They allow the library to include the library directly as a .framework or .a but they support and recommend specifying dependencies using Cocoapods which are then linked and installed by their CLI. They also use .xconfig files to specify build settings and allow merging of Info.plist entries. It's a very nice setup overall and leaves end users with basically nothing to do (if it all works correctly).

[mauron85]

I was considering same approaches, with exactly same reasoning (which is very encouraging).
Still not decided yet which one is the best, so we can discuss.

Requires each native library to be packaged, built, and distributed on NPM to be used. This would be fine for common libraries but would discourage use of smaller libraries.

Well, one of my goals is to provide tooling for third party developers to create (and publish) such a package easily. Most common packages (modules) could be published in @native_modules (or similar) scope to npm. Third party under their npm's "private" (not so) scope.

In mean time I've started work on react-native's local-cli linker/xcode project parser replacement. Even if decision will be made to go with Carthage, I think it's worth to be working on. The current one uses cordova xcode project and have some issues:

• doesn't allow to create project from scratch (only edit existing) - not big issue, just space for improvement
• modules can be added only to specific groups (workarounded in local-cli tool)
• links all targets (if project has multiple targets like static lib, framework - all will be linked into app project, which will throw clang's linker error) Because of this, I already see some 3-rd party developers distribute npm packages like CocoaLumberjack stripped to contain single target only - static lib.
• no documentation
• written to suit cordova needs

New linker/project parser will also add features like:

• printing dependency graph
• exporting/importing project from/to multiple formats (JSON, Graphviz? ... )
• written in typescript (with strict types)
• tested with unit tests

The linker/project parser will work nicely with carthage too, but doesn't make sense with cocoapods.

EDIT: edited multiple times

[mauron85]

They allow the library to include the library directly as a .framework or .a but they support and recommend specifying dependencies using Cocoapods which are then linked and installed by their CLI.

Reacting only to the first part of the sentence (cocoapods are different story).
The problem with bundling frameworks into package is that it will necessarily result to multiple packages bundling same framework (just matter of time). This will lead to clang's linker error "IOS build error duplicate symbols for architecture".

You really need some global repository of native modules, either it will be carthage or npm (or cocoapods). Even then there will be some conflicts coming from 3-rd party packages (but in much smaller scale - let's hope).

EDIT: edited multiple times.

[matt-oakes]

I edited my comment above with links to how NativeScript handles plugins with dependencies and custom build settings. They have some nice idea which we could borrow.

Well, one of my goals is to provide tooling for third party developers to create (and publish) such a package easily. Most common packages (modules) could be published in @native_modules (or similar) scope to npm. Third party under their npm scope.

That makes sense. Having them in NPM would still mean that new versions would need to be published after the native library itself publishes an update. We'd need to be careful to make sure that this isn't reliant on a small group of people to do it or for there to be a single point of failure where someone is the only person with access but they are no longer actively maintaining the package.

But probably main reason, I started work on npm variant is that I'm not very happy with react-native's local-cli linker anyway. Even if decision will be made to go with cocoapods/carthage, I think it's worth to be working on. The current one uses cordova xcode project and have some issues:

...

XCodeGen looks like a good solution to this. It takes a YAML or JSON file and generates XCode project files from it. The file can contain dependencies and supports Carthage out of the box.

I agree that the local-cli link is pretty bad at the minute. I actually manually link most of the time as it's broken often enough that I've found it's easier to just do it myself. My background is in native app development though, which isn't typical of the React Native community at large.

The problem with bundling frameworks into package is it will necessarily result to multiple packages bundling same framework. This will lead to clang's linker error "IOS build error duplicate symbols for architecture x86_64".

You really need some global repository of native modules, either it will be carthage or npm.

I've hit that issue as well. I even had the fun of one of the native libraries which React Native itself embeds clashing with an embedded one from a non-opensource Google library. That was very hacky to solve.

I believe this is why they strongly recommend using Cocoapods instead.

[mauron85]

XCodeGen looks like a good solution to this. It takes a YAML or JSON file and generates XCode project files from it.

Looks interesting. But don't see having it any sort of API. It looks like it only supports create project from initial configuration file. But might be wrong. Missing some API's that will update that file with configuration from package.json. Something like:

// read package json and link all native-modules (frameworks, libs,...)
const proj = XcodeProject.parse('project.pbxproject');
proj.getGroup('ReactNative').addFramework('New Framework');
proj.save('project.pbxproject');

EDIT: but it could be added easily, as it's matter of updating json file (much easier than write whole parser from scratch).

[matt-oakes]

Looks interesting. But don't see having any sort of API. It looks like it only supports create project from initial configuration file. But might be wrong. You still need some tool that it will update that file with some API, something like:

The idea would be to generate the xcodeproj files from the YAML or JSON using the XcodeGen tool. If you needed to make a change you would edit the YAML or JSON and then regenerate the xcodeproj file.

This has the advantage that the edits are very simple to make; you're just editing a JSON file with a known structure. However, it does mean that we're requiring another binary and completely deleting and regenerating the users XCode project file, which won't work if the user has made changes to it manually or if they're using React Native in an existing native app.

I agree that altering the project file directly is likely the best solution. The generator uses this Swift library to do it and there is also the Ruby library which Cocoapods uses. Neither of these are easy to integrate directly with a node script, but they seem to have much better support than the Cordova one which you mentioned.

[elicwhite]

Cc @grabbou. I think he said he wrote a parser and transformer for xcodeproj files for React-Native link.

[mauron85]

Combination of XcodeGen, Carthage and xcodeproj transformer seems like viable solution to me.

But how it will work, without any changes to core react-native? How will third party plugins link their native modules? So far I can think of, following steps would be necessary:

1. Transform xcode project to XcodeGen format
2. Read package.json native modules declaration (yet to be defined) and update some main carthage Cartfile
3. Run carthage update to update and install native dependencies
4. Also add dependencies (same as in step 2.) into XcodeGen project spec file
5. Transform and save XcodeGen project spec file back to xcode project

IMO without the react-native support every package would have to do all above steps (in postlink phase). 2. - 4. are fine, but what worries me are repeating steps 1. and 5.

We could get rid of those transformations steps (1. and 5.), but it would require some support from react-native itself. React-native would have to replace the pbxproject with XcodeGenFormat and wrote some scripts that will run step 5. as part of build step. But not sure how real it is (to ask rn team to implement/merge something like this).

More questions:

1. How will be XcodeGen, Carthage distributed? Or is there any chance, they become part of react-native itself? That would need strong cooperation with rn team. Or wil be packaged and published as npm packages?
2. How it will be different from cocoapods?

EDIT: My former proposal assumed no cooperation (no changes to core react-native) with rn team whatsoever. However if such cooperation is possible, then what we really need is this native module declaration, so every plugin can declare it's native dependencies in package.json and rn linker install and links them automatically (no matter if with cocoapods or carthage/XcodeGen), mitigating the "IOS build error duplicate symbols for architecture" issue.

[mauron85]

We'd need to be careful to make sure that this isn't reliant on a small group of people to do it or for there to be a single point of failure where someone is the only person with access but they are no longer actively maintaining the package.

100% agree. Less a problem when carthage is used. In matter of fact, I already commited some PRs to add carthage support for libraries that didn't have it.

matghazaryan/SOMotionDetector#35
ziminji/objective-c-sql-query-builder#34

The process of adding carthage support can be pretty much reduced to single command with https://github.com/mauron85/XcodeToolSource (uses XcodeTool and modified template)

[axe-fb]

Relevant discussion - #18

[grabbou]

Sorry I haven't taken part in this discussion earlier - I was a bit busy working on the CLI related things. This is a very important topic and something we need to figure out - not just the "best" way to handle native dependencies of 3rd party native modules, but native modules itself too.

Let me read through this issue and come up with some observations soon.