chore(dev): RFC for a tooling revamp

[ofek]

• rendered
• PoC

[netlify]

✅ Deploy Preview for vector-project canceled.

Name	Link
🔨 Latest commit	3d55611
🔍 Latest deploy log	https://app.netlify.com/sites/vector-project/deploys/636c8b08eb19cd0008c29ed6

[netlify]

❌ Deploy Preview for vrl-playground failed.

Name	Link
🔨 Latest commit	3d55611
🔍 Latest deploy log	https://app.netlify.com/sites/vrl-playground/deploys/636c8b08b1b9550008b61026

[spencergilbert]

I think it would be valuable to determine why we've decided to write this in Rust vs another language (likely Python). My surface level review would lead me to believe having it in Python would be easier for onboarding and ux as you wouldn't need to go through compiling it (and possibly getting deps to compile) before usage.

On the other hand, it would be difficult to develop against Vector without having the tools and know-how to compile vdev as well.

[ofek]

I added a note 👍

[tobz]

Overall, I really like this RFC in spirit.

I do see a very large practical problem with achieving the intended outcome, vis-á-vis only running tests that are "affected" by the PR, if we need to manually list out what changes should trigger the tests.

I'd personally still like to see this tooling even if it has to be invoked manually, and so I think it could be worth fleshing out/expanding what it would look like to use this to drive the two most common workflows: run all integration tests, and run a specific integration test. Essentially, use those two examples as part of the RFC and show more specific examples of invocation.

[jszwedko]

Generally like the direction here. In hindsight, I think this should have been two RFCs: one for the devtool, one for the integration test refactor, to focus the discussions around each aspect (granted the integration test refactor would depend on the devtool).

My biggest question is around developer productivity of having the tool be in Rust. I'm curious to get other team members' opinions here though.

[jszwedko]

I'm definitely 👍 on the dedicated CLI, but I did have some concern about using Rust for it since I was concerned that might decrease developer productivity when working within the CLI, which has less of a concern on correctness and does a lot of subprocess execution. It sounds like you've found it plenty productive though, given your experience working on a similar tool in Python? It does have the advantage that everyone on the team is experienced in Rust.

As another data point, @tobz how would you have felt writing https://github.com/vectordotdev/vector/blob/master/scripts/generate-components-docs.rb in Rust vs. Ruby?

[spencergilbert]

I also thought of this yesterday - if this is to power CI, how/when would it be built for CI?

If we need to compiler during the the CI run that adds time (hopefully not much because the code should be small...) or built only on changes to the tool and pull from those artifacts.

[tobz]

As another data point, @tobz how would you have felt writing generate-components-docs.rb in Rust vs. Ruby?

At the time, it probably would have sucked more to slog through writing it in Rust from scratch. Doing that now... probably wouldn't be terrible because it'd be easy to know when I reached feature parity by comparing the outputs directly, and having a good mental model of how it's structured.

🤷🏻

[lukesteensen]

My biggest question here is around the scope of the vdev tool and whether it's worthwhile to build something custom. There are a lot of options in between make (which I agree is quite awful) and writing something totally custom. Some examples I'm aware of are just and the cargo xtask pattern. Was anything like one of those considered?

For the integration tests specifically I can see a stronger argument for writing some tooling from scratch, but still it seems like the scope should be minimized.

On the Python question, I would strong prefer Rust over any language that:

1. Not everyone on the team has experience with
2. Brings in all kinds of additional setup issues like virtualenvs, etc

[ofek]

Some examples I'm aware of are just and the cargo xtask pattern. Was anything like one of those considered?

Yes just is basically the same as now but with Windows support and xtasks are in Rust like the dedicated tool but with added complexity of Cargo and lots of boilerplate

[lukesteensen]

I guess I should clarify that my comments are primarily about the vdev tool itself, not about refactoring our integration tests or building some better tooling around them. I agree with @jszwedko that this would be a lot easier to discuss if those two things were presented independently.

just is basically the same as now but with Windows support

This might just be showing my lack of imagination for what types of things vdev will be able to do. Looking at the PoC, I'm seeing a handful of different subcommands:

• build: This appears to simply wrap cargo build. Why would I run this instead of using cargo myself (which I already know how to do, is more flexible, and we won't have to teach new devs)?
• config: Deal with configuration of vdev itself, including repo which is just used for building paths for other commands and org/orgs which don't seemed to be used (other than the Starship prompt).
• exec: This seems to run an arbitrary command. It's not clear what value this adds over typing the same command without vdev exec in front of it.
• int: Again, building some nice tooling around our integration tests seems good 👍
• meta: The idea of a Starship integration is neat, but in the current one the crate version is already shown by default and the org isn't something that's used anywhere.
• status: Not sure why I'd run this over git status.

I understand it's just a proof of concept at this stage, but I can really only see the value in the integration test tooling. If we're going to build the rest of it, we should have some clearer justification of why it's more valuable than just using a better command runner and/or migrating scripts away from bash.

xtasks are in Rust like the dedicated tool but with added complexity of Cargo and lots of boilerplate

I'm not entirely sure what you mean by the "complexity" of Cargo, but the integration with Cargo is one of the main benefits of the pattern: you don't need to build and distribute a separate tool. Otherwise it seems largely similar to what you're doing with vdev, given that you're really just building an arbitrary program in Rust, so I don't see the boilerplate argument either. I'm not trying to argue that we should definitely use xtask, it just seems like we should justify in the RFC why we want to differ from an existing pattern in the Rust community that seems(?) to fit our problem.

---

Overall I do feel like I'm missing something here, so apologies if that's the case. A little bit more about how the tooling presented here is meant to address each item in the "Pain" section would probably help ensure we're all on the same page. Some of them are obvious (e.g. Windows support), but with others (e.g. test failures are hard to debug) it's a little less obvious.

[ofek]

• build: This does wrap cargo build. As discussed in standup half of the team uses Make which does the same things like smart selection of features
• config: This could determine org settings to send data, dev settings like scripts/features, etc.
• exec: As shown in the readme this always enters the root so you can be in any directory, a good accessibility feature too
• meta: These would be any beta or one-off scripts that need not clutter the root namespace
• status: This could show which components have changed and need testing, the progress of an ongoing release, etc.

---

I'm not entirely sure what you mean by the "complexity" of Cargo, but the integration with Cargo is one of the main benefits of the pattern: you don't need to build and distribute a separate tool.

This would require having Cargo and a compiler on every runner rather than fetching a single binary.

I don't see the boilerplate argument

That way requires modification of Vector's workspace + a special directory structure and config file. Maybe we're reading different docs?

[lukesteensen]

A lot of those example are cool and interesting things, but aren't quite at a level of solving the concrete pain points in this RFC. I'm certainly not against having them, but I don't think they alone are enough to justify writing a new tool right now.

This would require having Cargo and a compiler on every runner rather than fetching a single binary.

This is an important point I think. Is it a requirement to run these tools/tasks somewhere without a Rust toolchain? My assumption has been that it will be used in place where you're already assumed to be able to compile Vector itself.

That way requires modification of Vector's workspace + a special directory structure and config file. Maybe we're reading different docs?

We're already using a workspace, so it's ~3 lines of config and then writing a new crate just like vdev: #15191

---

To move this forward, I think I would recommend focusing for now on the concrete problems of (1) integration tests, and (2) Windows support. That would involve setting up either something independent like vdev or an xtask setup, whichever people think would work best, and building first for the integration test use case while also opening up a path to migrate the scripts directory into Rust over time. That way the scope is relatively bounded to resolving current higher priority issues, but we're free to expand the tool in the future as we see fit.

[ofek]

enough to justify writing a new tool right now

To be clear, what is your proposal? Also, are you aware that everything you asked for regarding testing is already implemented?

[ofek]

Also: https://github.com/ofek/vector/blob/tooling-rfc/rfcs/2022-10-31-15056-tooling-revamp.md#out-of-scope

[neuronull]

I know there is still discussion ongoing, and others on the team have more experience/history to weigh in with than me.
But the questions I had were answered so I'll drop in my approval (for what that's worth 😃 ).
As I mentioned in standup I think overall the pros outweigh the cons wrt to selecting Rust for the language.

[tobz]

Related to Luke's concern, and after reading some of the documentation for cargo-xtask: it feels to me like if we decided in the future that cargo-xtask had some benefit that we didn't fully appreciate at the time of approving this RFC, we could migrate to it.

It essentially has you create a crate that builds into a binary, and takes positional arguments, and so on. To me, this means that porting vdev over to cargo-xtask would be supremely simple. Beyond that, cargo-xtask doesn't appear to have much value beyond trying to establish cargo as the entrypoint command for all dev-related tasks, similar to what npm allows for.

Given that we don't currently use cargo-xtask, it seems like the de facto winner would be the tooling we already "have", being the PoC that Ofek has built already.

As far as Ofek's comments around being able to distribute a single binary: for local Vector development, pulling a single binary does feel slightly moot since you're already expecting to be able to compile Vector, so you'll have the toolchain to build vdev as well.

However, given that vdev might end up being used for doing other tasks (comparing configuration schemas is one thing I could imagine myself adding to it) where there's no need/desire to actually build Vector itself, I can see the value if having it be able to function as a standalone tool that could be distributed individually.

[tobz]

I'm a semi-firm "yes" on this.

I think, like many things, there's too many possible uses of this tool to properly categorize them all up front and figure out if this approach is the absolute best approach, and so on... but I agree with the approach insofar as it will certainly unify our fragmented ecosystem of scripts and commands used in local development and CI, and provide a more familiar base (Rust) to build further tooling on top of.

[spencergilbert]

Similarly soft-✅, it seems like a good step forward even if we change the actual implementation down the line the actual meat of the tool should be reusable.

[jszwedko]

@ofek is there anything else you want to do with this or can we merge it?

[ofek]

Merge away!

[bruceg]

Noting a late comment on this for posterity.

While I do think the overall tool idea could be a benefit to Vector development, I'm not a big fan of this particular mechanism for integration tests. I would anticipate a lot of shared code between these tests, leading to these crates being bogged down with a lot of boilerplate, as evidenced by the follow-up PRs to implement them.

Note that having one directory per integration test is likely a win to collect all configuration together, my objection is to the duplication and opportunities for one-off errors that this would induce.

[ofek]

Shared test utilities already noted in https://github.com/ofek/vector/blob/tooling-rfc/rfcs/2022-10-31-15056-tooling-revamp.md#future-improvements