TOML lens #91

raphink · 2014-03-06T13:32:48Z

TOML was brought to my attention several times recently: https://github.com/mojombo/toml

The attached patchset covers all the examples provided in the docs.

CAVEATS

QUOTES! Bare data types need to be clearly identified in order to force quoting only for values that are not integers, floats, booleans or datetimes… Even then, could a user want a quoted boolean? If so, we'll have to put quotes in the values, which is really ugly…

One way to fix this would be to type values strictly using subnodes for examples:

{ "foo" = "42" { "type" = "integer" } }
{ "bar" = "baz" { "type" = "string" } }

or simply marking them as quoted/bare:

{ "foo" = "bar" { "quoted" } }

or

{ "foo" = "true" { "bare" } }

tlimoncelli · 2015-05-19T18:37:16Z

Any update on this? I think a TOML parser would be very useful. Plus, I think the TOML project would benefit from having a lens because it assures that the language is regular.

raphink · 2015-05-19T19:26:16Z

@tlimoncelli do you have an opinion on the possible implementation choices outlined above?

tlimoncelli · 2015-05-19T20:36:30Z

I guess the decision comes down to: should the subnodes reflect the TOML standard, which says that each value has a type; or should it reflect how it is formatted (quoted or bare).

IMHO either is fine. I think the quoted/bare is sufficient; recording the type also works but seems to add detail that isn't used. Implement whatever is easiest.

I would make sure a tests that include:

foo = "false"
bar = "true"

as well as

foo = false
bar = true

There should also be a tests that set the value (to the string and to the boolean).

I do have a concern that the subnode could be confused with actual data if someone were to iterate over the data. Is that a possibility?

Does any other lens use this technique of creating subnotes to store metadata?

raphink · 2015-05-20T07:17:48Z

@tlimoncelli I currently have no use for that lens, but I'm willing to get it in shape. The current implementation parses the string and boolean values the same way. What would you suggest as a tree to keep the TOML functionalities?

tlimoncelli · 2015-05-20T14:54:29Z

I'm not entirely sure I understand the question but that rarely prevents me from answering :-)

At a high level the request is that the lens should retain quotes (or lack of quotes) when it parses foo = "true" or bar = true. The first is a string value, the second is a boolean.

Of the implementation proposals listed above, I think the best is to add subnodes that indicate whether the element was quoted or not. For example this file:

foo = "true"
bar = true

should create this tree:

{ "foo" = "true" { "quoted" } }
{ "bar" = "true" { "bare" } }

raphink · 2015-06-12T10:20:12Z

@tlimoncelli I see the issue. The tree should definitely denote the typing.

tlimoncelli · 2015-06-15T15:01:01Z

It has dawned on me that instead of "quoted" and "bare", better words might be "string" and "boolean".

{ "foo" = "true" { "string" } }
{ "bar" = "true" { "boolean" } }

tlimoncelli · 2015-08-25T13:38:11Z

Any update?

raphink · 2015-08-25T16:17:23Z

@tlimoncelli sorry, haven't looked at it for some time.

raphink · 2015-08-25T16:22:45Z

The thing with "string"/"boolean" is that it only fixes the issue for booleans, not general quoting.

raphink · 2015-08-26T07:27:11Z

I've just played around a little. If we want to reflect the typing, we will have to make a lens with a complex tree like Json.lns to avoid ambiguities.

descrepes · 2015-09-24T15:16:16Z

Hi,

I just play around with that lens but got some issues.
Here is the scenario with the "TOML Example" file :
(Just tried to modify the title)

augtool -A -t 'Toml incl /root/test.toml'
augtool> print /files/root/test.toml                                              
/files/root/test.toml
/files/root/test.toml/#comment = "This is a TOML document."
/files/root/test.toml/title = "TOML Example"
/files/root/test.toml/@table[1] = "owner"
/files/root/test.toml/@table[1]/name = "Tom Preston-Werner"
/files/root/test.toml/@table[1]/dob = "1979-05-27T07:32:00-08:00"
/files/root/test.toml/@table[1]/dob/#comment = "First class dates"
/files/root/test.toml/@table[2] = "database"
/files/root/test.toml/@table[2]/server = "192.168.1.1"
/files/root/test.toml/@table[2]/ports
/files/root/test.toml/@table[2]/ports/elem[1] = "8001"
/files/root/test.toml/@table[2]/ports/elem[2] = "8001"
/files/root/test.toml/@table[2]/ports/elem[3] = "8002"
/files/root/test.toml/@table[2]/connection_max = "5000"
/files/root/test.toml/@table[2]/enabled = "true"
/files/root/test.toml/@table[3] = "servers"
/files/root/test.toml/@table[3]/#comment = "Indentation (tabs and/or spaces) is allowed but not required"
/files/root/test.toml/@table[4] = "servers.alpha"
/files/root/test.toml/@table[4]/ip = "10.0.0.1"
/files/root/test.toml/@table[4]/dc = "eqdc10"
/files/root/test.toml/@table[5] = "servers.beta"
/files/root/test.toml/@table[5]/ip = "10.0.0.2"
/files/root/test.toml/@table[5]/dc = "eqdc10"
/files/root/test.toml/@table[6] = "clients"
/files/root/test.toml/@table[6]/data
/files/root/test.toml/@table[6]/data/elem[1]
/files/root/test.toml/@table[6]/data/elem[1]/elem[1] = "gamma"
/files/root/test.toml/@table[6]/data/elem[1]/elem[2] = "delta"
/files/root/test.toml/@table[6]/data/elem[2]
/files/root/test.toml/@table[6]/data/elem[2]/elem[1] = "1"
/files/root/test.toml/@table[6]/data/elem[2]/elem[2] = "2"
/files/root/test.toml/@table[6]/#comment = "Line breaks are OK when inside arrays"
/files/root/test.toml/@table[6]/hosts
/files/root/test.toml/@table[6]/hosts/elem[1] = "alpha"
/files/root/test.toml/@table[6]/hosts/elem[2] = "omega"
augtool> set /files/root/test.toml/title "totototot"
augtool> print /files/root/test.toml                                              
/files/root/test.toml
/files/root/test.toml/#comment = "This is a TOML document."
/files/root/test.toml/title = "totototot"
/files/root/test.toml/@table[1] = "owner"
/files/root/test.toml/@table[1]/name = "Tom Preston-Werner"
/files/root/test.toml/@table[1]/dob = "1979-05-27T07:32:00-08:00"
/files/root/test.toml/@table[1]/dob/#comment = "First class dates"
/files/root/test.toml/@table[2] = "database"
/files/root/test.toml/@table[2]/server = "192.168.1.1"
/files/root/test.toml/@table[2]/ports
/files/root/test.toml/@table[2]/ports/elem[1] = "8001"
/files/root/test.toml/@table[2]/ports/elem[2] = "8001"
/files/root/test.toml/@table[2]/ports/elem[3] = "8002"
/files/root/test.toml/@table[2]/connection_max = "5000"
/files/root/test.toml/@table[2]/enabled = "true"
/files/root/test.toml/@table[3] = "servers"
/files/root/test.toml/@table[3]/#comment = "Indentation (tabs and/or spaces) is allowed but not required"
/files/root/test.toml/@table[4] = "servers.alpha"
/files/root/test.toml/@table[4]/ip = "10.0.0.1"
/files/root/test.toml/@table[4]/dc = "eqdc10"
/files/root/test.toml/@table[5] = "servers.beta"
/files/root/test.toml/@table[5]/ip = "10.0.0.2"
/files/root/test.toml/@table[5]/dc = "eqdc10"
/files/root/test.toml/@table[6] = "clients"
/files/root/test.toml/@table[6]/data
/files/root/test.toml/@table[6]/data/elem[1]
/files/root/test.toml/@table[6]/data/elem[1]/elem[1] = "gamma"
/files/root/test.toml/@table[6]/data/elem[1]/elem[2] = "delta"
/files/root/test.toml/@table[6]/data/elem[2]
/files/root/test.toml/@table[6]/data/elem[2]/elem[1] = "1"
/files/root/test.toml/@table[6]/data/elem[2]/elem[2] = "2"
/files/root/test.toml/@table[6]/#comment = "Line breaks are OK when inside arrays"
/files/root/test.toml/@table[6]/hosts
/files/root/test.toml/@table[6]/hosts/elem[1] = "alpha"
/files/root/test.toml/@table[6]/hosts/elem[2] = "omega"
augtool> save
error: Failed to execute command
saving failed (run 'print /augeas//error' for details)
augtool> print /augeas//error
/augeas/files/root/test.toml/error = "put_failed"
/augeas/files/root/test.toml/error/path = "/files/root/test.toml"
/augeas/files/root/test.toml/error/lens = "/usr/share/augeas/lenses/dist/toml.aug:95.10-.45:"
/augeas/files/root/test.toml/error/message = "Failed to match \n    ({ /[A-Za-z][.0-9A-Z_a-z-]+/ = /([^]\\001-\\004\\t\\n\\r \"#,[](([^]\\001-\\004\\r#,=[]+)*[^]\\001-\\004\\t\\n\\r \"#,[])?)/ } | { /[A-Za-z][.0-9A-Z_a-z-]+/ = /([^]\\001-\\004\\n\\r\"[]*#+[^]\\001-\\004\\n\\r\"[]*)/ } | { /[A-Za-z][.0-9A-Z_a-z-]+/ } | { } | { /#comment/ = /[^\\001-\\004\\t\\n\\r ][^\\001-\\004\\n]*[^\\001-\\004\\t\\n\\r ]|[^\\001-\\004\\t\\n\\r ]/ })*({ /@table/ = /[^]\\001-\\004\\n\\r.]+(\\\\.[^]\\001-\\004\\n\\r.]+)*/ } | { /@@table/ = /[^]\\001-\\004\\n\\r.]+(\\\\.[^]\\001-\\004\\n\\r.]+)*/ })*\n  with tree\n    { \"#comment\" = \"This is a TOML document.\" } {  } { \"title\" = \"totototot\" } {  } { \"@table\" = \"owner\" } { \"@table\" = \"database\" } { \"@table\" = \"servers\" } { \"@table\" = \"servers.alpha\" } { \"@table\" = \"servers.beta\" } { \"@table\" = \"clients\" }"

I'm using augeas 1.2.

Any ideas ?

Thx a lot :)

jrabbit · 2016-08-20T03:08:31Z

any update on this?

flyinbutrs · 2017-03-09T21:06:23Z

Any movement on this? @raphink?

raphink · 2017-03-13T11:55:54Z

@jrabbit @flyinbutrs no, I haven't had a use for this (although I do use TOML in other contexts). If someone wants to work on it, I'd welcome patches.

TODO: inline tables

raphink · 2018-10-16T13:35:19Z

@tlimoncelli @descrepes @jrabbit @flyinbutrs I've just restarted work on this, using the Json (fully typed) approach. Afaik it's only missing inline table support and quotes/dots on keys 🤢, so testing/feedback is most welcome!

lutter · 2018-10-18T17:09:48Z

Is this enough to parse your typical Cargo.toml ? If so, I'd be all for committing this now and refining it later.

raphink · 2018-10-18T21:56:54Z

Do you have a full example @lutter?

raphink · 2018-10-19T09:00:11Z

At least https://github.com/slog-rs/example-lib/blob/master/Cargo.toml parses properly:

$ augtool -At "Toml.lns incl /tmp/cargo.toml" -I lenses/ print /files
/files
/files/tmp
/files/tmp/cargo.toml
/files/tmp/cargo.toml/table[1] = "package"
/files/tmp/cargo.toml/table[1]/entry[1] = "name"
/files/tmp/cargo.toml/table[1]/entry[1]/string = "slog-example-lib"
/files/tmp/cargo.toml/table[1]/entry[2] = "version"
/files/tmp/cargo.toml/table[1]/entry[2]/string = "0.3.0"
/files/tmp/cargo.toml/table[1]/entry[3] = "authors"
/files/tmp/cargo.toml/table[1]/entry[3]/array
/files/tmp/cargo.toml/table[1]/entry[3]/array/string = "Dawid Ci\304\231\305\274arkiewicz <[email protected]>"
/files/tmp/cargo.toml/table[1]/entry[4] = "description"
/files/tmp/cargo.toml/table[1]/entry[4]/string = "Example of a library utilizing slog-rs"
/files/tmp/cargo.toml/table[1]/entry[5] = "keywords"
/files/tmp/cargo.toml/table[1]/entry[5]/array
/files/tmp/cargo.toml/table[1]/entry[5]/array/string[1] = "log"
/files/tmp/cargo.toml/table[1]/entry[5]/array/string[2] = "logging"
/files/tmp/cargo.toml/table[1]/entry[5]/array/string[3] = "structured"
/files/tmp/cargo.toml/table[1]/entry[5]/array/string[4] = "hierarchical"
/files/tmp/cargo.toml/table[1]/entry[5]/array/string[5] = "example"
/files/tmp/cargo.toml/table[1]/entry[6] = "license"
/files/tmp/cargo.toml/table[1]/entry[6]/string = "MPL-2.0"
/files/tmp/cargo.toml/table[1]/entry[7] = "documentation"
/files/tmp/cargo.toml/table[1]/entry[7]/string = "https://dpc.github.io/slog-rs/"
/files/tmp/cargo.toml/table[1]/entry[8] = "homepage"
/files/tmp/cargo.toml/table[1]/entry[8]/string = "https://github.com/dpc/slog-rs"
/files/tmp/cargo.toml/table[1]/entry[9] = "repository"
/files/tmp/cargo.toml/table[1]/entry[9]/string = "https://github.com/dpc/slog-rs"
/files/tmp/cargo.toml/table[1]/entry[10] = "readme"
/files/tmp/cargo.toml/table[1]/entry[10]/string = "README.md"
/files/tmp/cargo.toml/table[2] = "lib"
/files/tmp/cargo.toml/table[2]/entry = "path"
/files/tmp/cargo.toml/table[2]/entry/string = "lib.rs"
/files/tmp/cargo.toml/table[3] = "dependencies"
/files/tmp/cargo.toml/table[3]/entry[1] = "slog"
/files/tmp/cargo.toml/table[3]/entry[1]/string = "2"
/files/tmp/cargo.toml/table[3]/entry[2] = "slog-stdlog"
/files/tmp/cargo.toml/table[3]/entry[2]/string = "3"
/files/tmp/cargo.toml/table[4] = "dev-dependencies"
/files/tmp/cargo.toml/table[4]/entry[1] = "env_logger"
/files/tmp/cargo.toml/table[4]/entry[1]/string = "0.3.5"
/files/tmp/cargo.toml/table[4]/entry[2] = "slog-term"
/files/tmp/cargo.toml/table[4]/entry[2]/string = "2"
/files/tmp/cargo.toml/table[4]/entry[3] = "slog-async"
/files/tmp/cargo.toml/table[4]/entry[3]/string = "2"

raphink · 2018-10-19T09:01:25Z

Note: in principle, arrays don't allow mixing types for elements (I don't really see why btw), but this lens allows it.

raphink · 2018-11-09T06:52:50Z

@lutter do you want to merge this as is?

raphink · 2018-11-29T10:23:15Z

I'll just merge as it is.

lutter · 2018-11-29T19:53:18Z

Thanks for doing that ! THat's what I've been meaning to do, too.

raphink · 2018-11-30T07:35:36Z

There's still a few details to fix imo, I'll try to run it on as many TOML files as possible and open issues.

* fix(lens): toml: add write support The toml lens fails to save simple toml files due to problems with multi dimensional (>2) arrays. Therefore disable arrays with more than 2 dimensions for now to enable basic usage again. This commit fixes issue #715 by applying the suggested workaround. If anyone can come up with a better solution please feel free to do so. Fixes: 5e1cd31 (TOML lens (#91), 2018-11-29) Signed-off-by: Richard Leitner <[email protected]> * fix(lens): toml: update reference link toml.io now provides a versioned reference URL, therefore use this instead of the old GitHub repo link. Signed-off-by: Richard Leitner <[email protected]> * fix(lens): toml: typo in About: License Signed-off-by: Richard Leitner <[email protected]> * test(lens): toml: add basic write This test covers the bug described in issue #715. Signed-off-by: Richard Leitner <[email protected]> * test(lens): toml: fix "array of arrays" The "Test: Toml.array_norec; Array of arrays" tested a 3 dimensional array. This commit fixes the test to actually test a 2 dimensional array. More than 2 dimensional arrays are currently not supported due to issue #715. Signed-off-by: Richard Leitner <[email protected]> Co-authored-by: George Hansper <[email protected]>

raphink self-assigned this Mar 1, 2014

raphink added the improvement label Apr 16, 2014

raphink force-pushed the dev/toml branch 2 times, most recently from 1ae2414 to e890a47 Compare May 19, 2015 20:06

jfilak mentioned this pull request Jun 5, 2018

question: tool for adding insecure registries containers/skopeo#515

Closed

raphink added 2 commits October 16, 2018 13:26

Toml: New lens to parse the TOML format

1f0d685

TOML: fix bare_re regexp

b15dd2c

raphink force-pushed the dev/toml branch from 066e13f to b15dd2c Compare October 16, 2018 11:26

Take Json approach to TOML lens

60c0ab0

TODO: inline tables

Support inline_tables

c0a8790

raphink merged commit 5e1cd31 into hercules-team:master Nov 29, 2018

g0hl1n mentioned this pull request Oct 26, 2021

fix(lens): toml: write support #742

Merged

TOML lens #91

TOML lens #91

Uh oh!

Conversation

raphink commented Mar 6, 2014

CAVEATS

Uh oh!

tlimoncelli commented May 19, 2015

Uh oh!

raphink commented May 19, 2015

Uh oh!

tlimoncelli commented May 19, 2015

Uh oh!

raphink commented May 20, 2015

Uh oh!

tlimoncelli commented May 20, 2015

Uh oh!

raphink commented Jun 12, 2015

Uh oh!

tlimoncelli commented Jun 15, 2015

Uh oh!

tlimoncelli commented Aug 25, 2015

Uh oh!

raphink commented Aug 25, 2015

Uh oh!

raphink commented Aug 25, 2015

Uh oh!

raphink commented Aug 26, 2015

Uh oh!

descrepes commented Sep 24, 2015

Uh oh!

jrabbit commented Aug 20, 2016

Uh oh!

flyinbutrs commented Mar 9, 2017

Uh oh!

raphink commented Mar 13, 2017

Uh oh!

raphink commented Oct 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lutter commented Oct 18, 2018

Uh oh!

raphink commented Oct 18, 2018

Uh oh!

raphink commented Oct 19, 2018

Uh oh!

raphink commented Oct 19, 2018

Uh oh!

raphink commented Nov 9, 2018

Uh oh!

raphink commented Nov 29, 2018

Uh oh!

lutter commented Nov 29, 2018

Uh oh!

raphink commented Nov 30, 2018

Uh oh!

Uh oh!

raphink commented Oct 16, 2018 •

edited

Loading