Support consuming and producing nuget packages with native binaries

[davidfowl]

We need a convention for both consuming and adding native binaries to nuget packages.

When you're developing a project and need to take a dependency on a native binary, you need to put it in a path that is loadable. This scenario needs to take bitness and OS into account.

You also need to be able to produce a package that has these native binaries inside of it.

[monoman]

Finally someone echoes my 3-years old issue: see
https://nuget.codeplex.com/workitem/1221

Rafael Teixeira
O..:.)oooo

On Tue, Jul 1, 2014 at 1:32 AM, David Fowler notifications@github.com
wrote:

We need a convention for both consuming and adding native binaries to
nuget packages.

When you're developing a project and need to take a dependency on a native
binary, you need to put it in a path that is loadable. This scenario needs
to take bitness and OS into account.

—
Reply to this email directly or view it on GitHub
#402.

[davidfowl]

Data points:

Here are some existing projects using native dependencies:

• Kestrel
• DataCommon.SQLite
• Helios (today helios has a native project in the same solution as the managed project)

Kestrel has binaries committed to source control in this format:

$/src/Microsoft.AspNet.Server.Kestrel/native/win/x86
$/src/Microsoft.AspNet.Server.Kestrel/native/win/amd64
$/src/Microsoft.AspNet.Server.Kestrel/native/darwin/universal

It also keeps the same structure in the nupkg itself:

https://github.com/aspnet/KestrelHttpServer/blob/dev/build/_custom-goals.shade#L12

The build script above puts the native binaries at the root of the nuget package with the same layout.

At runtime, it loads the native library using the following method:

https://github.com/aspnet/KestrelHttpServer/blob/e4b9bd265c75704529409638fd9cdfac504a93ef/src/Microsoft.AspNet.Server.Kestrel/KestrelEngine.cs#L26

https://github.com/aspnet/KestrelHttpServer/blob/dev/src/Microsoft.AspNet.Server.Kestrel/Networking/Libuv.cs#L23

Ideally, with the KRuntime, we should have conventions around what directories we automatically add for loading native binaries (we the option of specifying this in the project.json). Of course the library author can always manually load library, but that only really works if you can package all native dependencies in the nuget package. This doesn't work well on *nix, see aspnet/KestrelHttpServer#10 for more details.

SqlLite is a bit more interesting:

The build process downloads sqlite dynamically and for packaging, it uses an msbuild script on net451 so that nuget will automagically copy the binaries into the bin folder of the project.

https://github.com/aspnet/DataCommon.SQLite/blob/dev/makefile.shade#L46
https://github.com/aspnet/DataCommon.SQLite/blob/dev/src/Microsoft.Data.SQLite/build/net451/Microsoft.Data.SQLite.targets

At runtime it uses load library (similar to kestrel):

https://github.com/aspnet/DataCommon.SQLite/blob/dev/src/Microsoft.Data.SQLite/Utilities/NativeLibraryLoader.cs#L16

[davidfowl]

@lodejard and @GrabYourPitchforks Need your POV on this. Assuming we invent a convention for determining where native assets get loaded from within a package based on OS and Bitness, we'll either modify PATH or LD_LIBRARY_PATH.

[GrabYourPitchforks]

Thou shalt not modify the environment block (including environment variables) while the application is running. It's not yours to mess with.

[davidfowl]

Another option might be to call LoadLibrary (and dlopen on *nix) on the appropriate binary. The only downside to this might be that we would load the library eagerly instead of lazily.