[RFC] River: A Flow-optimized config language #1839

rfratto · 2022-06-29T17:53:53Z

This RFC expands on the expressions design proposed by #1546 with a suggestion that a custom config language for Flow expressions is required to make Flow be as useful and powerful as possible. The current implementation of Flow uses HCL, but with the number of changes I had to make to HCL to get things to work, it doesn't appear to be the best fit.

I would prefer not creating a new language, but I see it as our only option for Flow. Most of this RFC focuses on the rationale for how I reached this conclusion, why existing languages aren't the best fit, and why it's important to do this for Flow earlier instead of later.

Additionally, in light of proposing a new language, I focus on reducing the learning curve as much as possible to reduce friction from users migrating from the existing YAML configs to Flow.

The current prototype of River is in the main branch, but is subject to removal if we abandon the proposal.

docs/rfcs/xxxx-river.md

marctc · 2022-06-30T09:15:08Z

docs/rfcs/xxxx-river.md

+
+### Why now?
+
+Grafana Agent Flow is already a dramatic change to the Agent. To avoid users


Is this new language necessary for an MVP? I was thinking if HCL is enough for the scope of the MVP, should we invest time on design a new language if later users wouldn't be happy to adopt Flow?

docs/rfcs/xxxx-river.md

tpaschalis · 2022-07-04T07:57:27Z

docs/rfcs/xxxx-river.md

+* Make it easy for developers to create Flow components which operate with
+  arbitrary Go values (interfaces, channels, etc.)
+* Expose error messages in an easily understandable and actionable way
+* Natively support using Go values of any type in config expressions


It's the fourth time we're referencing 'arbitrary Go values' in the summary so we definitely consider it important to us. We mention 'constructing data pipelines', but do we have any more concrete explanation to make apparent why someone would need to pass a channel, an interface or a complex Go struct around?

I feel like I'm already covering this when I first brought up the term:

Values can refer to arbitrary Go values like interfaces or channels, enabling component developers to easily allow users to construct data pipelines using Go APIs without requiring knowledge of the underlying implementation.

Is there a way I could make this clearer?

docs/rfcs/xxxx-river.md

This commit introduces the lexer for River (grafana#1839). It is not wired in anywhere, and is the first of several PRs to slowly bring over functionality from the prototype. Note that this commit does not indicate we will ship River, just that we are continuing the prototyping process in the main branch. pkg/river may still be removed in the future in favor of an alternative solution.

This commit introduces the lexer for River (grafana#1839). It is not wired in anywhere, and is the first of several PRs to slowly bring over functionality from the prototype. Most of the scanner code has been copied from go/scanner, adapted slightly for River and decorated with extra comments. Note that this commit does not indicate we will ship River, just that we are continuing the prototyping process in the main branch. pkg/river may still be removed in the future in favor of an alternative solution.

* pkg/river: introduce scanner package This commit introduces the lexer for River (#1839). It is not wired in anywhere, and is the first of several PRs to slowly bring over functionality from the prototype. Most of the scanner code has been copied from go/scanner, adapted slightly for River and decorated with extra comments. Note that this commit does not indicate we will ship River, just that we are continuing the prototyping process in the main branch. pkg/river may still be removed in the future in favor of an alternative solution. * pkg/river: fix lint errors * pkg/river: hush unparam warnings for Scanner.switch2 Assuming `=` as the next rune seems harder to follow than explicitly passing it, even if it's always the same argument. * address review feedback * river/scanner: fix grammar for float Float should allow exponents on numbers without a `.` * river/scanner: denote that sign after exponent is optional

This commit introduces the parser for River (grafana#1839). It is not wired in anywhere, and is the second of several PRs to slowly bring over functionality from the prototype. The parser code has been highly inspired by go/parser, with tests copied from the go/parser tests. Note that this commit does not indicate we will ship river, just that we are continuing the prototyping process in the main branch. pkg/river may still be removed in the future in favor of an alternative solution for Flow.

* pkg/river: introduce parser package This commit introduces the parser for River (#1839). It is not wired in anywhere, and is the second of several PRs to slowly bring over functionality from the prototype. The parser code has been highly inspired by go/parser, with tests copied from the go/parser tests. Note that this commit does not indicate we will ship river, just that we are continuing the prototyping process in the main branch. pkg/river may still be removed in the future in favor of an alternative solution for Flow. * update golangci-lint * river: fix bugs found by parser fuzzer * update Go and golanglint-ci versions in Drone for Windows * update SHA for golangci-lint * windows ci: only update Go since the Windows job doesn't do any linting * .gitattributes: disable autocrlf and force \n line endings for fuzz tests * wip: experiment with using a golang windows container for windows tests * Revert "wip: experiment with using a golang windows container for windows tests" This reverts commit a32b59d. * river/parser: include comments when parsing * river/parser: add missing ) in error message

This commit introduces the first pass at a value package for River (grafana#1839). It is not wired in anywhere yet, and further PRs will improve the package and eventually wire it in. Note that this commit does not indicate we will ship River, just that we are continuing the prototyping process in the main branch. pkg/river may still be removed in the future in favor of an alternative solution for Flow. This commit purposefully leaves three larger tasks incomplete to be done in follow up commits: 1. Ability to invoke function values 2. Proper encoding/decoding to Go structs with rvr block tags (currently, labels are ignored) 3. Decoding to Go structs with missing required attributes should fail

Structs are always objects unless they implement the capsule marker interface.

marctc · 2022-08-09T15:14:08Z

@rfratto we probably should merge this before releasing any flow version, wdyt?

rfratto · 2022-08-09T20:18:51Z

@rfratto we probably should merge this before releasing any flow version, wdyt?

Yeah, agreed, I'll open it up for final review now.

mattdurham

Ship it

rfratto force-pushed the rfc-river branch from 06da5f8 to 23fd621 Compare June 29, 2022 17:57

rfratto requested a review from mattdurham June 29, 2022 18:07

rfratto changed the title ~~[RFC] River: Flow-specific configuration language~~ [RFC] River: A Flow-optimized config language Jun 29, 2022

mattdurham reviewed Jun 29, 2022

View reviewed changes