feat(jsonnet): Environment vendor (#198)

* feat(jsonnet): tkrc.yaml as root marker

Adds new root marker (`tkrc.yaml`) alongside `jsonnetfile.json` to be
able to mark the actual project root when environment-local vendoring is used.

* doc: document vendor overriding

Author: tombrk

docs/docs/directory-structure.md

@@ -8,9 +8,9 @@ route: /directory-structure
 Tanka uses the following directories and special files:
 
 ```bash
-. # the project
+. # the project (<rootDir>)
 ├── environments # code defining clusters
-│   └── default
+│   └── default # <baseDir>
 │       ├── main.jsonnet # starting point of the Jsonnet compilation
 │       └── spec.json # environment's config
 ├── jsonnetfile.json # direct dependencies
@@ -26,51 +26,55 @@ Tanka uses the following directories and special files:
 ```
 
 ## Environments
+
 Tanka organizes configuration in environments. For the rationale behind this,
 see the [section in the tutorial](/tutorial/environments).
 
 An environment consists of at least two files:
 
 #### spec.json
+
 This file configures environment properties such as cluster connection
 (`spec.apiServer`), default namespace (`spec.namespace`), etc.
 
 For the full set of options, see the [Golang source
 code](https://github.com/grafana/tanka/blob/master/pkg/spec/v1alpha1/config.go).
 
 #### main.jsonnet
+
 Like other programming languages, Jsonnet needs an entrypoint into the
 evaluation, something to begin with. `main.jsonnet` is exactly this: The very
 first file being evaluated, importing or directly specifying everything required
 for this specific environment.
 
+## Root and Base
 
-## Libraries
-Tanka builds on code-reuse, by refactoring common pieces into libraries, which can be imported from two locations:
+When talking about directories, Tanka uses the following terms:
 
-### lib
-The `lib/` folder is for libraries that are meant for only this single project.
-If you intend to deploy your custom e-commerce stack, you could for example have
-libraries for the `auth`, `bookings`, `billing` and `inventory` here.
+| Term      | Description                              | Identifier file                   |
+| --------- | ---------------------------------------- | --------------------------------- |
+| `baseDir` | The root of your project                 | `jsonnetfile.json` or `tkrc.yaml` |
+| `rootDir` | The directory of the current environment | `main.jsonnet`                    |
 
-They are not intended to be shared and thus are a good fit for `lib/`
+Regardless what subdirectory of the project you are in, Tanka will always be
+able to identify both directories, by searching for the identifier files in the
+parent directories.  
+Tanka needs these for correctly setting up the [import paths](/libraries/import-paths).
 
-> **Note:** Opposing to `vendor/`, `lib/` is entirely your realm. You manage the
-> contents and Tanka won't ever mess with this after `tk init`.
+This is similar to how `git` always works, by looking for the `.git` directory.
 
-### vendor
-Some libraries can be useful to many projects (for example ones for
-[Prometheus](https://prometheus.io), [Loki](https://grafana.com/loki), etc).
+## Libraries
+
+Tanka relies heavily on code-reuse, so libraries are a natural thing. Roughly
+spoken, they can be imported from two paths:
 
-These are usually published on GitHub. To use them in your project, [install
-them using `jb`](/libraries/install-publish#install-a-library). This will store
-a copy of the source code on the remote in the `vendor/` directory. Note that
-this folder belongs to `jb` and all files not recorded in
-`jsonnetfile.lock.json` will be removed on the next run. Also don't edit files
-in here (your changes would be reverted anyways), use
-[Shadowing](/libraries/import-paths#shadowing) instead.
+- `/lib`: Project local libraries
+- `/vendor` External libraries
+
+For more details consider the [import paths](/libraries/import-paths).
 
 ### jsonnetfile.json and the lock
+
 `jb` records all external packages installed in a file called
 `jsonnetfile.json`. This file is the source of truth about what should be
 included in `vendor/`. However, it should only include what is really directly

docs/docs/libraries/import-paths.md

@@ -9,23 +9,14 @@ menu: Libraries
 When using `import` or `importstr`, Tanka considers the following directories to
 find a suitable file for that specific import:
 
-1. `<baseDir>`: The directory of your environment (`/environments/default`,
-   etc). Put things that only belong to a single environment here.
-2. `/lib`: Libraries created for this very project, not meant to be shared
-   otherwise. Put everything you need across multiple environments here.
-3. `/vendor`: Shared libraries installed using `jsonnet-bundler`. Do not modify
-   this folder by hand, your changes will be overwritten by `jb` anways.
+| Rank | Path               | Purpose                                                                                                                      |
+| ---- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------- |
+| 4    | `<baseDir>`        | The directory of your environment, e.g. `/environments/default`.<br /> Put things that belong to this very environment here. |
+| 3    | `/lib`             | Project-global libraries, that are used in multiple environments, but are specific to this project.                          |
+| 2    | `<baseDir>/vendor` | Per-environment vendor, can be used for [`vendor` overriding](/libraries/overriding#per-environment)                         |
+| 1    | `/vendor`          | Global vendor, holds external libraries installed using `jb`.                                                                |
 
-> **Note**: The directories are visited in the above order. For example, when a
-> file is present in both, `/lib` and `/vendor`, the one from `/lib` will be
-> taken, as it occurs higher in the list.
-
-### Shadowing
-It is possible to shadow certain files (overlay them with another version), by
-putting a file with the exact same name and into a higher ranked import path.
-This can be handy if you need to do temporary changes to a vendored library by
-overlaying the to-be-changed files using new ones in `lib/`.
-
-For example, to shadow `/vendor/my/lib/file.libsonnet`, copy it to
-`/lib/my/lib/file.libsonnet` and do your changes. Tanka will take the file in
-`lib/` instead of `vendor/` from now on.
+> **Note**:
+>
+> - If a file occurs in multiple paths, the one with the highest rank will be chosen.
+> - `/` in above table means `<rootDir>`, which is your project root.

docs/docs/libraries/overriding.md

@@ -0,0 +1,75 @@
+---
+name: Overriding
+route: /libraries/overriding
+menu: Libraries
+---
+
+# Overriding vendor
+
+The `vendor` directory is immutable in it's nature. You can't and should never
+modify any files inside of it, `jb` will revert those changes on the next run anyways.
+
+Nevertheless, it can sometimes become required to to changes there, e.g. if an
+upstream library contains a bug that needs to be fixed immediately, without
+waiting for the upstream maintainer to review it.
+
+## Shadowing
+
+Because [import paths](/libraries/import-paths) are ranked in Tanka, you can use
+a technique called shadowing: By putting a file with the exact same name in a
+higher ranked path, Tanka will prefer that file instead of the original in
+`vendor`, which has the lowest possible rank of 1.
+
+For example, if `/vendor/foo/bar.libsonnet` contained an error, you could create
+`/lib/foo/bar.libsonnet` and fix it there.
+
+> **Tip:** Instead of copying the file to the new location and making the edits,
+> use a relative import and [patching](/tutorial/environments#patching):
+>
+> ```jsonnet
+> // in /lib/foo/bar.libsonnet:
+> import "../vendor/foo/bar.libsonnet" + {
+>   foo+: {
+>     bar: "fixed"
+>   }
+> }
+> ```
+
+## Per environment
+
+Another common case is overriding the entire `vendor` bundle per environment.
+
+This is handy, when you for example want to test a change of an upstream
+library which is used in many environments (including `prod`) in a single one,
+without affecting all the others.
+
+For this, Tanka let's you have a separate `vendor`, `jsonnetfile.json` and
+`jsonnetfile.lock.json` per environment. To do so:
+
+#### Create `tkrc.yaml`
+
+Tanka normally uses the `jsonnetfile.json` from your project to find it's root.
+As we are going to create another one of that down the tree in the next step, we
+need another marker for `<rootDir>`.
+
+For that, create an empty file called `tkrc.yaml` in your project's root,
+alongside the original `jsonnetfile.json`.
+
+> **Info**: While the name suggests that `tkrc.yaml` could be used for setting
+> parameters, this is not case yet.  
+> It might however be repurposed later, in case we need such functionality
+
+#### Add a `vendor` to your environment
+
+In your environments folder (e.g. `/environments/default`):
+
+```bash
+# init jsonnet bundler (creates jsonnetfile.json)
+$ jb init
+
+# install the updated dependency
+$ jb init github.com/foo/bar@v2
+```
+
+> **Tip**: You don't need to install everything into the new `vendor/`, as
+> packages not present there can still be imported from the global `/vendor`.

docs/doczrc.js

@@ -40,6 +40,7 @@ export default {
         // "Using libraries",
         // "Creating and structure",
         "Installing and publishing",
+	"Overriding",
       ],
     },
     "Command-line completion",

pkg/jsonnet/jpath/dirs_test.go

@@ -0,0 +1,145 @@
+package jpath
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+
+	"io/ioutil"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+type scenario struct {
+	name        string
+	testdata    []string
+	environment string
+
+	// expected results
+	baseDir string
+	rootDir string
+	err     error
+}
+
+// TestFindRoot asserts that baseDir and rootDir can be correctly resolved.
+//
+// To do so, Tanka searches the directory tree from the passed directory up twice:
+// - for main.jsonnet to find the baseDir
+// - for jsonnetfile.json (or tkrc.yaml) to find the rootDir
+//
+// This enables a git-like behaviour (regardless how deep you are in the
+// project, it works)
+func TestFindRoot(t *testing.T) {
+	scenarios := []scenario{
+		// Scenario: Missing base pointerfile. We expect an ErrorNoBase.
+		{
+			name:        "missing-basePointer",
+			environment: "environments/default",
+			testdata:    []string{"jsonnetfile.json", "environments/default/"},
+			err:         ErrorNoBase,
+		},
+		// Scenario: Missing root pointerfile. We expect an ErrorNoRoot.
+		{
+			name:        "missing-rootPointer",
+			environment: "environments/default",
+			testdata:    []string{"environments/default/main.jsonnet"},
+			err:         ErrorNoRoot,
+		},
+
+		// Make sure jsonnetfile.json works as a pointer
+		scenarioPointer("jsonnetfile.json"),
+		// Make sure tkrc.yaml works as a pointer
+		scenarioPointer("tkrc.yaml"),
+
+		// Per-environment vendoring is tricky, because environments get their
+		// own `jsonnetfile.json`, so `rootDir` would yield the same thing as
+		// `baseDir`, which is usually not wanted.
+		//
+		// Scenario 1: No tkrc.yaml to mark the actual root. `baseDir` and
+		// `rootDir` should be equal
+		scenarioLocalVendor(false),
+		// Scenario 2: A tkrc.yaml is added o the actual root. `rootDir` should
+		// be the actual root, `baseDir` the environment.
+		scenarioLocalVendor(true),
+	}
+
+	for _, s := range scenarios {
+		require.NotZero(t, s.environment)
+
+		t.Run(s.name, func(t *testing.T) {
+			dir := makeTestdata(t, s.testdata)
+			defer os.RemoveAll(dir)
+
+			_, base, root, err := Resolve(filepath.Join(dir, s.environment))
+			assert.Equal(t, s.err, err)
+
+			if err == nil {
+				assert.Equal(t, filepath.Join(dir, s.baseDir), base)
+				assert.Equal(t, filepath.Join(dir, s.rootDir), root)
+			}
+		})
+	}
+}
+
+func scenarioLocalVendor(tkrc bool) scenario {
+	name := "localvendor"
+	td := []string{
+		"jsonnetfile.json",
+		"environments/default/main.jsonnet",
+		"environments/default/jsonnetfile.json",
+	}
+	// first jsonnetfile.json is in baseDir, so it will become rootDir as well
+	root := "environments/default"
+
+	if tkrc {
+		name += "-with-tkrc"
+		td = append(td, "tkrc.yaml") // add tkrc
+		// now root should be project_root instead
+		root = "/"
+	}
+
+	return scenario{
+		name:        name,
+		environment: "environments/default",
+		testdata:    td,
+
+		rootDir: root,
+		baseDir: "environments/default",
+	}
+}
+
+func scenarioPointer(ptr string) scenario {
+	return scenario{
+		name:        "pointer-" + ptr,
+		environment: "environments/default",
+
+		testdata: []string{
+			"environments/default/main.jsonnet",
+			ptr,
+		},
+
+		baseDir: "environments/default",
+		rootDir: "/",
+	}
+}
+
+func makeTestdata(t *testing.T, td []string) string {
+	t.Helper()
+
+	tmp, err := ioutil.TempDir("", "tk-dirsTest")
+	require.NoError(t, err)
+
+	for _, f := range td {
+		dir, file := filepath.Split(f)
+		if dir != "" {
+			err := os.MkdirAll(filepath.Join(tmp, dir), os.ModePerm)
+			require.NoError(t, err)
+		}
+		if file != "" {
+			_, err := os.Create(filepath.Join(tmp, dir, file))
+			require.NoError(t, err)
+		}
+	}
+	return tmp
+}