Skip to content

Commit 3a5e906

Browse files
Yohe-Amzifeodestifo
authored
feat!: envs module (#46)
* feat(envs): Envs module * feat: rework `ghjk.ts` interface and integrate with other modules * wip: minor fixup * fix: improve uninited dir experience * wip: `$ ghjk envs sync` * refactor(envs,cli)!: envs first cli (#48) * refactor(envs): envs first CLI * wip: missing file * wip: wip * wip: wip 2 * feat(tests): basic tests * docs: some snippets for `README.md` * doc: typo * fix: `Deno.Command` troubles * fix: cross platform shell fn for getting ctime * fix: `!` as instId separator * fix(tests): missing flag * wip: wip * feat: `ghjk p resolve` * refactor: polish CLI code * wip: clearEnv fix * fix: vendor dax patch * fix: forgotten change * fix: use `@ghjk/dax` fork * fix: remove vendor dir from Dockerfile * feat: env hooks (#59) * feat(envs,tasks): env `onEnter`/`onExit` hooks * wip: anon tasks * feat: anonymous tasks * tests: anon tasks * wip: wip * feat: env onEnter/onExit hooks * tests: env hook tests * fix: bug in path outputs * fix: use latest setup-ghjk * fix: bug on check task * chore: bump deno to 1.43.1 * fix: miss field in test harness * fix: use `no-prune` on docker tests * fix: don't rmi at all * fix: timeout * docs: metatype ecosystem banner in README.md (#49) * docs: metatype ecosystem banner in README.md * fix: utm --------- Co-authored-by: Teo Stocco <[email protected]> * fix(port): `cpy_bs` doesn't work on arm (#60) * wip: linux/arm in docker on mac-14 * wip: try `custom-macos` * wip: try inline docker * wip: try `custom-macos` * fix: remove unnecessary action * fix: move platform flags into tests * refactor: merge new job into matrix * fix: broken aarch64 ports * wip: log * fix: diambiguate platforms in harness * fix: silly ci bug * fix: env depending on itself case * fix: lint issue * fix: use 5m timeout for everything * refactor(envs): `GHJK_NEXTFILE` based env reloading (#83) * wip: nextfile * tests: nextfile * refactor: pid based nextfile * refactor: replace with ghjkdir vars instead of $HOME * fix: env hook tests * fix: bashisms * feat(modules): ports outdated command (#62) * wip(modules): add installSetIds to * wip(modules-ports): read recipe.json and get cooked installSetIds * wip(ports): wip for version getter func * feat(modules): restrcutre ctx tree and recipe.json content * feat(ports): add table to show version diff * fix: fix unhandled prov type in envs/posix cook * wip: wip update outdated installs * wip: add ghjk file with multiple installs * chore(test): wip add test * refactor: refactor table structure and update cargo_binstall * refactor(ghjk.ts): replace `secureConfig` with `hack.ts` (#87) * refactor(ghjk.ts): replace secureConfig with hack.ts * docs: improve README * fix(ci): update action job * fix: address llm feedback * fix: cargo-binstall bug * fix: ports outdated test fix * feat!: metatype integration polish (#91) * feat: multipl env inheritance * refactor: handle GHJK_CLEANUP_POSIX edgecases * feat: task on task inheritance * fix: bugs! * feat(port): `deno_ghrel` * fix: `portsOutdated` tests * fix: diamond inheritance bug * small stuff * fix: apply llm feedback * fix: minor fixes * fix: path var merging * feat: `std.ts` and `sedLock` * fix: task path combinations * fix: task path combinations 2 * chore: set version 0.2.0 * fix: more target sed pattern * fix(ci): pre-commit issue * fix: 10m timeout * fix: explicit GITHUB_TOKEN --------- Co-authored-by: Teo Stocco <[email protected]> Co-authored-by: Estifanos Bireda <[email protected]>
1 parent d46ddd4 commit 3a5e906

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

98 files changed

+7757
-2752
lines changed

.ghjk/deno.lock

Lines changed: 253 additions & 136 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

.ghjk/lock.json

Lines changed: 508 additions & 539 deletions
Large diffs are not rendered by default.

.github/workflows/nightly.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ on:
44
workflow_dispatch:
55

66
env:
7-
DENO_VERSION: "1.42.1"
7+
DENO_VERSION: "1.44.2"
88
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
99
GHJK_LOG_PANIC_LEVEL: error
1010
DENO_DIR: .deno-dir
@@ -55,7 +55,7 @@ jobs:
5555
runs-on: ubuntu-latest
5656
steps:
5757
- uses: actions/checkout@v4
58-
- uses: metatypedev/setup-ghjk@2e8bbf084060a18828338a7cdd43fde6feb2a3cc
58+
- uses: metatypedev/setup-ghjk@318209a9d215f70716a4ac89dbeb9653a2deb8bc
5959
with:
6060
installer-url: ./install.ts
6161
env:

.github/workflows/tests.yml

Lines changed: 15 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -9,11 +9,13 @@ on:
99
- ready_for_review
1010

1111
env:
12-
DENO_VERSION: "1.42.1"
13-
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
12+
DENO_VERSION: "1.44.2"
1413
GHJK_LOG: debug
1514
GHJK_LOG_PANIC_LEVEL: error
1615
DENO_DIR: .deno-dir
16+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
17+
# removing the images after every test is unncessary
18+
DOCKER_NO_RMI: 1
1719

1820
jobs:
1921
changes:
@@ -30,7 +32,14 @@ jobs:
3032
- uses: denoland/setup-deno@v1
3133
with:
3234
deno-version: ${{ env.DENO_VERSION }}
35+
# run ghjk once to avoid trigger file changes when
36+
# pre commit runs ghjk. We'll always see changes
37+
# to lock.json since GITHUB_TOKEN is different
38+
# in the CI
39+
- run: deno run --unstable -A main.ts print config
3340
- uses: pre-commit/[email protected]
41+
env:
42+
SKIP: ghjk-resolve
3443

3544
test-e2e:
3645
runs-on: "${{ matrix.os }}"
@@ -40,7 +49,7 @@ jobs:
4049
- os: ubuntu-latest
4150
platform: linux/x86_64
4251
e2eType: "docker"
43-
- os: custom-macos
52+
- os: custom-arm
4453
platform: linux/aarch64
4554
e2eType: "docker"
4655
- os: macos-latest
@@ -77,12 +86,12 @@ jobs:
7786
runs-on: ubuntu-latest
7887
steps:
7988
- uses: actions/checkout@v4
80-
- uses: metatypedev/setup-ghjk@2e8bbf084060a18828338a7cdd43fde6feb2a3cc
89+
- uses: metatypedev/setup-ghjk@318209a9d215f70716a4ac89dbeb9653a2deb8bc
8190
with:
8291
installer-url: ./install.ts
8392
env:
8493
GHJKFILE: ./examples/protoc/ghjk.ts
8594
- run: |
86-
cd examples/protoc
95+
cd examples/tasks
8796
. $(ghjk print share-dir-path)/env.sh
88-
protoc --version
97+
ghjk x hey

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,3 @@
11
.DS_Store
22
play.*
3+
examples/**/.ghjk

.pre-commit-config.yaml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,16 @@ repos:
3535
- commit-msg
3636
- repo: local
3737
hooks:
38+
- id: lock-sed
39+
name: Sed lock
40+
language: system
41+
entry: bash -c 'deno run --unstable -A main.ts x lock-sed'
42+
pass_filenames: false
43+
- id: ghjk-resolve
44+
name: Ghjk resolve
45+
language: system
46+
entry: bash -c 'deno run --unstable -A main.ts p resolve'
47+
pass_filenames: false
3848
- id: deno-fmt
3949
name: Deno format
4050
language: system

README.md

Lines changed: 162 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -11,73 +11,187 @@ ghjk /jk/ is a programmable runtime manager.
1111
1212
## Features
1313

14-
- install and manage tools (e.g. rustup, deno, node, etc.)
15-
- [ ] fuzzy match the version
16-
- support dependencies between tools
17-
- [ ] setup runtime helpers (e.g. pre-commit, linting, ignore, etc.)
18-
- [ ] provide a general regex based lockfile
19-
- enforce custom rules
20-
- [ ] create aliases and shortcuts
21-
- `meta` -> `cargo run -p meta`
22-
- `x meta` -> `cargo run -p meta` (avoid conflicts and provide autocompletion)
23-
- [ ] load environment variables and prompt for missing ones
24-
- [ ] define build tasks with dependencies
25-
- `task("build", {depends_on: [rust], if: Deno.build.os === "Macos" })`
26-
- `task.bash("ls")`
27-
- [x] compatible with continuous integration (e.g. github actions, gitlab)
14+
- Soft-reproducable developer environments.
15+
- Install posix programs from different backend like npm, pypi, crates.io.
16+
- Tasks written in typescript.
17+
- Run tasks when entering/exiting envs.
2818

2919
## Getting started
3020

3121
```bash
3222
# stable
33-
curl -fsSL https://raw.githubusercontent.com/metatypedev/ghjk/main/install.sh | bash
23+
curl -fsSL https://raw.githubusercontent.com/metatypedev/ghjk/0.2.0/install.sh | bash
3424
# latest (main)
35-
curl -fsSL https://raw.githubusercontent.com/metatypedev/ghjk/main/install.sh | GHJK_VERSION=main bash
25+
curl -fsSL https://raw.githubusercontent.com/metatypedev/ghjk/0.2.0/install.sh | GHJK_VERSION=main bash/fish/zsh
3626
```
3727

38-
In your project, create a configuration file `ghjk.ts`:
28+
In your project, create a configuration file called `ghjk.ts` that look something like:
3929

4030
```ts
41-
export { ghjk } from "https://raw.githubusercontent.com/metatypedev/ghjk/main/mod.ts";
42-
import * as ghjk from "https://raw.githubusercontent.com/metatypedev/ghjk/main/mod.ts";
43-
import node from "https://raw.githubusercontent.com/metatypedev/ghjk/main/ports/node.ts";
31+
// NOTE: All the calls in your `ghjk.ts` file are ultimately modifying the 'sophon' proxy
32+
// object exported here.
33+
// WARN: always import `hack.ts` file first
34+
export { sophon } from "https://raw.githubusercontent.com/metatypedev/ghjk/0.2.0/hack.ts";
35+
import {
36+
install, task,
37+
} from "https://raw.githubusercontent.com/metatypedev/ghjk/0.2.0/hack.ts";
38+
import node from "https://raw.githubusercontent.com/metatypedev/ghjk/0.2.0/ports/node.ts";
39+
40+
// install programs (ports) into your env
41+
install(
42+
node({ version: "14.17.0" }),
43+
);
44+
45+
// write simple scripts and execute them using
46+
// `$ ghjk x greet`
47+
task("greet", async ($, { argv: [name] }) => {
48+
await $`echo Hello ${name}!`;
49+
});
50+
```
51+
52+
Use the following command to then access your environment:
53+
54+
```bash
55+
ghjk sync
56+
```
57+
58+
### Environments
59+
60+
Ghjk is primarily configured through constructs called "environments" or "envs" for short.
61+
They serve as recipes for making (mostly) reproducable posix shells.
62+
63+
```ts
64+
export { sophon } from "https://raw.githubusercontent.com/metatypedev/ghjk/0.2.0/hack.ts";
65+
import * as ghjk from "https://raw.githubusercontent.com/metatypedev/ghjk/0.2.0/hack.ts";
66+
import * as ports from "https://raw.githubusercontent.com/metatypedev/ghjk/0.2.0/ports/mod.ts";
67+
68+
// top level `install`s go to the `main` env
69+
ghjk.install(ports.protoc());
70+
ghjk.install(ports.rust());
71+
72+
// the previous block is equivalent to
73+
ghjk.env("main", {
74+
installs: [
75+
ports.protoc(),
76+
ports.rust(),
77+
],
78+
});
79+
80+
ghjk.env("dev", {
81+
// by default, all envs are additively based on `main`
82+
// pass false here to make env independent.
83+
// or pass name(s) of another env to base on top of
84+
inherit: false,
85+
// envs can specify posix env vars
86+
vars: { CARGO_TARGET_DIR: "my_target" },
87+
installs: [
88+
ports.cargobi({ crateName: "cargo-insta" }),
89+
ports.act(),
90+
],
91+
})
92+
// use env hooks to run code on activation/deactivation
93+
.onEnter(ghjk.task(($) => $`echo dev activated`))
94+
.onExit(ghjk.task(($) => $`echo dev de-activated`));
95+
96+
ghjk.env({
97+
name: "docker",
98+
desc: "for Dockerfile usage",
99+
// NOTE: env references are order-independent
100+
inherit: "ci",
101+
installs: [
102+
ports.cargobi({ crateName: "cargo-chef" }),
103+
ports.zstd(),
104+
],
105+
});
106+
107+
// builder syntax is also availaible
108+
ghjk.env("ci")
109+
.var("CI", "1")
110+
.install(
111+
ports.opentofu_ghrel(),
112+
);
113+
114+
// each task describes it's own env as well
115+
ghjk.task({
116+
name: "run",
117+
inherit: "dev",
118+
fn: () => console.log("online"),
119+
});
120+
```
121+
122+
Once you've configured your environments:
123+
124+
- `$ ghjk envs cook $name` to reify and install an environment.
125+
- `$ ghjk envs activate $name` to switch to an environment.
126+
- And **most** usefully, `$ ghjk sync $name` to cook and _then_ activate an
127+
environment.
128+
- If shell is already in the specified env, it only does cooking.
129+
- Make sure to `sync` or `cook` your envs after changes.
130+
- If no `$name` is provided, most of these commands will operate on the default
131+
or currently active environment.
132+
133+
### Ports
134+
135+
TBD: this feature is in development.
136+
Look in the [kitchen sink](./examples/kitchen/ghjk.ts) for what's currently implemented.
44137

45-
ghjk.install(node({ version: "14.17.0" }));
138+
### Tasks
139+
140+
TBD: this feature is still in development.
141+
Look in the [tasks example](./examples/tasks/ghjk.ts) for what's currently implemented.
142+
143+
#### Anonymous tasks
144+
145+
Tasks that aren't give names cannot be invoked from the CLI.
146+
They can be useful for tasks that are meant to be common dependencies of other tasks.
147+
148+
### `hack.ts`
149+
150+
The imports from the `hack.ts` module, while nice and striaght forward to use, hold and modify global state.
151+
Any malicious third-party module your ghjkfile imports will thus be able to access them as well, provided they import the same version of the module.
152+
153+
```ts
154+
// evil.ts
155+
import { env, task } from "https://.../ghjk/hack.ts";
156+
157+
env("main")
158+
// lol
159+
.onEnter(task($ => $`rm -rf --no-preserve-root`);
46160
```
47161
48-
## How it works
49-
50-
The only required dependency is `deno`. Everything else is managed automatically
51-
and looks as follows (abstracting away some implementation details):
52-
53-
- the installer sets up a directory hook in your shell profile
54-
- `.bashrc`
55-
- `.zshrc`
56-
- `.config/fish/config.fish`
57-
- for every visited directory, the hook looks for `$PWD/ghjk.ts` in the
58-
directory or its parents, and
59-
- adds the `$HOME/.local/share/ghjk/envs/$PWD/shims/{bin,lib,include}` to your
60-
paths
61-
- sources environment variables in
62-
`$HOME/.local/share/ghjk/envs/$PWD/loader.{sh,fish}` and clear previously
63-
loaded ones (if any)
64-
- you can then
65-
- sync your runtime with `ghjk ports sync` which
66-
- installs the missing tools at `$HOME/.local/share/ghjk/ports/installs`
67-
- regenerates the shims with symlinks and environment variables
68-
- detects any violation of the enforced rules
69-
- [ ] `ghjk ports list`: list installed tools and versions
70-
- [ ] `ghjk ports outdated`: list outdated tools
71-
- [ ] `ghjk ports cleanup`: remove unused tools and versions
72-
73-
## Extending `ghjk`
162+
To prevent this scenario, the exports from `hack.ts` inspect the call stack and panic if they detect more than one module using them.
163+
This means if you want to spread your ghjkfile across multiple modules, you'll need to use functions described below.
164+
165+
> [!CAUTION]
166+
> The panic protections of `hack.ts` described above only work if the module is the first import in your ghjkfile.
167+
> If a malicious script gets imported first, it might be able to modify global primordials and get around them.
168+
> We have more ideas to explore on hardening Ghjk security.
169+
> This _hack_ is only a temporary compromise while Ghjk is in alpha state.
170+
171+
The `hack.ts` file is only optional though and a more verbose but safe way exists through...
74172
75173
```ts
174+
import { file } from "https://.../ghjk/mod.ts";
175+
176+
const ghjk = file({
177+
// items from `config()` are availaible here
178+
defaultEnv: "dev",
76179

180+
// can even directly add installs, tasks and envs here
181+
installs: [],
182+
});
183+
184+
// we still need this export for this file to be a valid ghjkfile
185+
export const sophon = ghjk.sophon;
186+
187+
// the builder functions are also accessible here
188+
const { install, env, task, config } = ghjk;
77189
```
78190
191+
If you intend on using un-trusted third-party scripts in your ghjk, it's recommended you avoid `hack.ts`.
192+
79193
## Development
80194
81195
```bash
82-
cat install.sh | GHJK_INSTALLER_URL=$(pwd)/install.ts bash
196+
$ cat install.sh | GHJK_INSTALLER_URL=$(pwd)/install.ts bash/fish/zsh
83197
```

check.ts

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,17 @@
11
#!/bin/env -S ghjk deno run --allow-env --allow-run --allow-read --allow-write=.
2-
// # FIXME: find a way to resolve !DENO_EXEC_PATH in shebangs
32

43
import "./setup_logger.ts";
54
import { $ } from "./utils/mod.ts";
65

76
const files = (await Array.fromAsync(
87
$.path(import.meta.url).parentOrThrow().expandGlob("**/*.ts", {
98
exclude: [
9+
".git",
1010
"play.ts",
1111
".ghjk/**",
1212
".deno-dir/**",
13+
"vendor/**",
14+
".git/**", // was throwing an error without this
1315
],
1416
}),
1517
)).map((ref) => ref.path.toString());

deno.jsonc

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,22 +2,36 @@
22
"tasks": {
33
"test": "deno test --parallel --unstable-worker-options --unstable-kv -A tests/*",
44
"cache": "deno cache deps/*",
5-
"check": "deno run -A check.ts"
5+
"check": "deno run -A ./check.ts"
66
},
77
"fmt": {
88
"exclude": [
9+
"*.md",
910
"**/*.md",
1011
".ghjk/**",
11-
".deno-dir/**"
12+
".deno-dir/**",
13+
"vendor/**"
1214
]
1315
},
1416
"lint": {
1517
"exclude": [
1618
".deno-dir/**",
1719
"ghjk.ts",
18-
"play.ts"
20+
"play.ts",
21+
"vendor/**"
1922
],
2023
"rules": {
24+
"include": [
25+
"no-console",
26+
"no-sync-fn-in-async-fn",
27+
"no-external-import",
28+
"no-inferrable-types",
29+
"no-self-compare",
30+
"no-throw-literal"
31+
// "verbatim-module-syntax"
32+
// "no-await-in-loop"
33+
// "ban-untagged-todo"
34+
],
2135
"exclude": [
2236
"no-explicit-any"
2337
]

0 commit comments

Comments
 (0)