YAML 2.0

[pwhelan]

YAML format proposal for Fluent-bit 2.0

In the interest of stream-lining the current YAML support to make it more
consistent as well as to make sure it supports the entire current feature
set of the current INI based configuration file format I propose the
following changes:

• Unnest the configuration map for plugins, ie: input, output and filters
  so that they are an array of maps with a name key for the plugin type.
• Allow defining property values as an array when using LISTS, f.e.:
  'record' for record_modifier.
• Allow the use of the 'include' property in plugin definitions so they
  can be processed in such a way to make it possible to replicate the
  behaviour of >=1.9.

Unnest plugin configuration map

At the moment the configuration for pipelines uses a map with the properties
input, filter and output. These maps then contain an array of maps with
a single key which acts as the plugin name which is a map of properties, ie:

pipeline:
    inputs:
        - dummy:
            rate: 5

To more closely fit in line with the expectations of most users the format
should use an array of properties, where the name property defines the plugin
type, f.e. :

pipeline:
    inputs:
        - name: dummy
          rate: 5

This would also simplify the parsing of the YAML configuration file format.

Allow defining properties as arrays

While this might require a bit more work, to properly support some plugins
it is necessary to be able to define the values of some properties as arrays.
The INI-based format simply defined a property multiple times to achieve this.
To do so in YAML we can simply define the value as an array.

This will increase the complexity of the parser in fluent-bit, unfortunately.

An example:

pipeline:
  inputs:
    - name: dummy
      rate: 5
  filters:
    - name: record_modifier
      record:
        - powered_by calyptia
        - version 0.0.1

The parser can easily identify when it is either a single scalar value or an
array and act accordingly by also inferring the correct behaviour using the
plugin's config map. This will become more complicated if the first
proposal is accepted and the name is processed after any other properties...

One way around this is would be to save all the property values and process
them at the end of the section.

Preserve the order of includes

*** This feature has not been approved. If anyone can give examples documentation could be prepared to help users who wish to migrate. ***

With the manner in which the actual pipeline is defined it is impossible
to reproduce the ordering of filters, which are based on load order, at
least if a pipeline is defined inside the main configuration file.

Resolving this problem is much more difficult, and in fact may simply be
(decided to be) to dificult to attempt to solve. With the legacy of issues
that have arisen due to this at least it should be addressed in some form.

One way of solving this would be to change the pipeline into an array so
it could include interspersed declarations of:

• include
• input
• filter
• output

Example

pipeline:
    - include: "settings.conf"
    - inputs:
        - name: dummy:
          rate: 5
    - include: "wooooot..."
    - filters:
        - name: record_modifier:
          record:
              - powered_by calyptia
              - version 0.0.1

This sort of structure would reinforce the idea of load time ordering for
filters.

As it stands it would be falling into the type of being similar
to the past structure used for the definition of each plugin where it
basically used a single key map to define a plugin.... Not good...
But at the very least it does resemble the syntax that tasks use
for ansible. Which could be a bad thing... Docker-compose does
no such nesting for declaring services in its format.

Examples for each config_map type:

Here is an example, interleaved with comments, showing how to represent each of the config_map types:

pipeline:
  - inputs:
      - name: example
        # FLB_CONFIG_MAP_STR
        # a string without quotes
        strdoublequotes: A Bare FLB_CONFIG_MAP_STR
        # a string with single quotes
        strdoublequotes: 'A Quoted STRING'
        # a string with double quotes
        strdoublequotes: "A double Quoted String"
        # FLB_CONFIG_MAP_STR_PREFIX
        # prefixed strings are no different ...
        strprefix: prefixed.str
        # FLB_CONFIG_MAP_INT
        # these are the formats supported by yaml: https://yaml.org/spec/1.2.2/#10213-integer
        plainint: 1
        negativeint: -3
        # FLB_CONFIG_MAP_BOOL
        # YAML uses either true or false
        booltrue: true
        boolfalse: false
        # FLB_CONFIG_MAP_DOUBLE
        # and doubles are numbers with dots...
        doubletrouble: 0.2
        # FLB_CONFIG_MAP_SIZE
        # .. will need to be a string that is handled internally by our own parser
        sizeplain: 1M
        sizequotes: "12G"
        # FLB_CONFIG_MAP_TIME
        # will need to do the same
        timeplain: 2H
        # FLB_CONFIG_MAP_CLIST
        # is nothing special and can be a simple string
        clist: path1,/path2,/path/3
        # not even attempting with FLB_CONFIG_MAP_MULT.
        # FLB_CONFIG_MAP_SLIST_1
        # with FLB_CONFIG_MAP_MULT:
        slists2:
          - example1
          - example2
        # FLB_CONFIG_MAP_SLIST_2
        # without FLB_CONFIG_MAP_MULT:
        slist2: property value
        # with FLB_CONFIG_MAP_MUTL:
        slists2:
          - another example
          - more examples

References:

• https://www.educba.com/ansible-include/
• https://docs.ansible.com/ansible/latest/collections/ansible/builtin/include_module.html
• https://docs.ansible.com/ansible/2.4/playbooks_reuse_includes.html

[nicolasparada]

In the "Preserve the order of includes" section, what prevents you from doing:

pipeline:
    - include: "settings.conf"
      inputs:
        - name: dummy:
          rate: 5
      filters:
        - name: record_modifier:
          record:
              - powered_by calyptia
              - version 0.0.1

One array item with all those properties?

[pwhelan]

In the "Preserve the order of includes" section, what prevents you from doing One array item with all those properties?

Then you could only include a single one and the order of when you include that include could also be indeterminate.

Or we could forget about includes completely and be something users do using templating.

I am somewhat of a fan of leaning on YAML to do the templating but we would have to make sure it is supported by libyaml. We would also then need to document it. The last time I used templating I used it just to not repeat similar values across a single file.

I'm thinking something like how kubernetes and helm work together to add templating.

As far as I know helm uses go templates or something similar and then mixes the files in an application specific method.

Or we could add a new fluent-bit filter plugin which takes a path and loads all of them as filters...

Personally I am not entirely against the idea, in principle. I believe filters are the only load order dependent plugins, but do not quote me on that. Those includes would have to include only filters and not inputs or outputs though.

[nicolasparada]

CC @edsiper

[nicolasparada]

I heard that fluent-bit doesn't even warranties the order of filters 👀

[pwhelan]

I think they do now, or there was an attempt at doing so since it was causing a lot of people problems.

[edsiper]

filters are always processed in order.