Merge branch 'mono/dev' into ptree

Author: cobycloud

pkgs/jaml/README.md

@@ -0,0 +1,125 @@
+# MEP-000: Introduction and Overview
+
+## 1. Abstract
+
+This proposal introduces our new markup language—a configuration format designed to be simple, human‐readable, and expressive. Drawing inspiration from established formats like TOML, YAML, and JSON, our language extends their core concepts with enhanced support for dynamic evaluation, explicit type annotations, and rich structural constructs. MEP-000 outlines the overarching goals, design principles, and key features that will guide the subsequent enhancement proposals (MEPs) for this language.
+
+---
+
+## 2. Motivation
+
+### 2.1. Simplicity and Readability
+
+- **Human-Readable Syntax:**  
+  Our language aims to maintain the clarity and simplicity of TOML, ensuring that configuration files remain accessible and easy to edit by humans.
+  
+- **Minimalistic Design:**  
+  The syntax is designed to be lightweight and unambiguous, reducing the learning curve while still providing powerful capabilities.
+
+### 2.2. Enhanced Expressiveness
+
+- **Static and Dynamic Constructs:**  
+  In addition to standard static data types, the language supports dynamic evaluation. This enables configurations to adapt based on external context (such as environment variables) or computed values.
+  
+- **Scoped Variables:**  
+  Variables can be defined and referenced within multiple scopes—global, table-local (self), and context-specific—allowing for flexible configuration strategies.
+
+### 2.3. Structural Flexibility
+
+- **Hierarchical Organization:**  
+  Inspired by TOML, our language supports sections (tables), table arrays, inline tables, and lists of inline tables, enabling users to organize configurations in a modular and logical manner.
+  
+- **Multiline Preservation:**  
+  Multiline strings, arrays, and inline tables retain their formatting, ensuring that embedded documentation, code snippets, or lengthy texts remain exactly as authored.
+
+### 2.4. Interoperability and Extensibility
+
+- **Mapping to Existing Standards:**  
+  The language is designed to map neatly to existing formats such as TOML, JSON, and YAML, facilitating integration with current tools and workflows.
+  
+- **Future-Proofing:**  
+  The design anticipates future enhancements such as dynamic evaluation of expressions, integration with external schema definitions, and advanced type checking.
+
+---
+
+## 3. Key Features
+
+### 3.1. Data Types and Structures
+
+- **Scalar Types:**  
+  Support for strings (including multiline strings), integers (decimal, octal, hexadecimal, binary), floats (with special values like `inf` and `nan`), booleans, and null.
+
+- **Collection Types:**  
+  Arrays (inline and multiline), inline tables, standard tables (sections), and table arrays, as well as lists of inline tables.
+
+- **Type Annotations:**  
+  Optional annotations provide clarity on expected data types and serve as documentation for validation and tooling.
+
+### 3.2. Dynamic Evaluation
+
+- **Evaluation Enclosures:**  
+  The language distinguishes between static values and dynamically evaluated expressions. Dynamic constructs allow the configuration to adapt based on runtime or external context.
+
+- **Evaluation Timing:**  
+  Support for load-time, render-time, and mixed evaluation modes ensures that users can precisely control when expressions are resolved.
+
+### 3.3. Scoped Variables
+
+- **Global Scope:**  
+  Variables defined in a global section, referenced with a designated symbol (e.g., `@{...}`), are available throughout the configuration.
+
+- **Self (Table-Local) Scope:**  
+  Variables defined within a table are resolved locally, allowing overrides of global defaults.
+
+- **Context Scope:**  
+  Variables provided externally (via context during rendering) are referenced with a specific notation (e.g., `${...}`) and enable dynamic configuration.
+
+### 3.4. File and Git Referencing
+
+- **File Referencing:**  
+  The language can incorporate external file content through explicit syntax, enabling modular configuration management.
+
+- **Git Referencing:**  
+  Integration with Git repositories allows for version-controlled configurations and dynamic inclusion of external content.
+
+### 3.5. Multiline and Preservation Features
+
+- **Multiline Strings:**  
+  Support for triple-quote syntax preserves newlines, indentation, and formatting exactly as written.
+
+- **Multiline Arrays and Inline Tables:**  
+  Arrays and inline tables may span multiple lines, preserving both order and readability.
+
+- **List of Inline Tables:**  
+  Inline tables can be used as elements within an array to represent lists of dictionaries, enhancing modular data representation.
+
+---
+
+## 4. Comparison with Existing Formats
+
+Our markup language draws on the strengths of formats like TOML by:
+- Maintaining a simple, clean syntax.
+- Supporting rich data types and hierarchical structures.
+- Extending these capabilities with dynamic evaluation and explicit type annotations.
+- Ensuring that all multiline and complex structures retain their original formatting.
+
+---
+
+## 5. Proposed MEP Roadmap
+
+To fully realize the language's capabilities, subsequent MEPs will address specific feature areas:
+- **MEP-001:** Supported Data Types and Structural Elements
+- **MEP-002:** Support for Scoped Variables (Global, Self, Context)
+- **MEP-003:** Multiline Constructs and Preservation
+- **MEP-004:** Type Annotations and Validation
+- **MEP-005:** Dynamic Evaluation and Expression Handling
+- **MEP-006:** File Referencing and Embedding
+- **MEP-007:** Git Referencing and Integration
+- *Additional MEPs will cover conditional logic, external schema integration, and more.*
+
+---
+
+## 6. Conclusion
+
+MEP-000 establishes the overall vision and design goals of our new markup language. It outlines a language that is both simple and powerful—maintaining the readability of TOML while extending its functionality to support dynamic evaluation, explicit type annotations, and a rich set of structural constructs. This foundational document provides the framework upon which the subsequent MEPs will build, ensuring a coherent and extensible specification.
+

pkgs/jaml/docs/MEP-0000 [A New Markup Language).md

@@ -0,0 +1,263 @@
+# MEP-001: Keywords, Operators, and Patterns
+
+## 1. Abstract
+
+This proposal specifies the reserved keywords, operators, punctuation, and common expression patterns for our markup language. It establishes clear rules for keyword usage, operator precedence, and syntax validation. Key updates include support for pipeline operations, enhanced string literal handling (with various quoting forms), conditional (ternary) expressions, membership and merge operators, and reserved functions like `File()` and `Git()`.
+
+---
+
+## 2. Motivation
+
+- **Consistency and Clarity:**  
+  Defining keywords, operators, and punctuation prevents naming conflicts and ambiguous syntax, ensuring consistent parsing and predictable expression evaluation.
+
+- **Avoiding Conflicts:**  
+  Reserving critical language constructs (keywords, functions, and punctuation) stops users from unintentionally overwriting core elements, which could lead to errors.
+
+- **Enhanced Error Handling:**  
+  A well-defined syntax with reserved elements enables robust parsing and clear error messages, aiding debugging and future tooling integrations.
+
+---
+
+## 3. Specification
+
+### 3.1. Reserved Keywords
+
+The following identifiers are reserved and **cannot** be used as names for keys, tables, variables, or user-defined types:
+
+- **Logical and Boolean Keywords:**  
+  - `true`, `false`, `null`, `is`, `not`, `and`, `or`
+- **Control Flow Keywords:**  
+  - `if`, **`elif`**, `else`, `for`, `in`
+- **Directive Keywords:**  
+  - `include`
+- **Special Functions:**  
+  - `File()`, `Git()`
+
+#### Keyword Usage
+- Keywords must appear in the appropriate context (for example, `if` and `elif` in conditional expressions).
+- Using any of these reserved words as identifiers will result in a syntax error.
+
+---
+
+### 3.2. Reserved Punctuations and Brackets
+
+#### 3.2.1. Reserved Punctuation Marks
+
+The following punctuation marks are reserved and **must not** be used as part of key names, variable names, or unquoted strings:
+
+- **Colon (`:`)** – For type annotations and key-value separation.
+- **Dot (`.`)** – For namespace access.
+- **Comma (`,`)** – As a separator in lists and inline tables.
+- **Equals Sign (`=`)** – For assignment in key-value pairs.
+- **Arrow (`->`)** – Reserved for potential functional expressions.
+- **Semicolon (`;`)** – Reserved for future extensions.
+- **Tilde (`~`)** – Used for load-time expressions.
+- **At Symbol (`@`)** – Global scope indicator.
+- **Dollar Sign (`$`)** – Context scope indicator.
+- **Percent Sign (`%`)** – Self (table-local) scope indicator.
+- **Less Than (`<`)** and **Greater Than (`>`)** – Used in comparisons.
+- **Exclamation Mark (`!`)** – For negation.
+- **Asterisk (`*`)** – For multiplication and comprehensions.
+- **Caret (`^`)** – Reserved for future use.
+- **Ampersand (`&`)** – Reserved for future use.
+- **Pipe (`|`)** – **Used as a pipeline operator** to chain operations.
+- **Forward Slash (`/`)** – In path strings and division.
+- **Backslash (`\`)** – As an escape character in strings.
+
+#### 3.2.2. Reserved Bracket Combinations
+
+All bracket types are reserved to ensure unambiguous syntax:
+
+- **Round Brackets (`()`)** – For grouping expressions and function calls.
+- **Square Brackets (`[]`)** – For defining lists and accessing array elements.
+- **Curly Braces (`{}`)** – For inline tables and dictionary-like structures.
+- **Angle Brackets (`<>`)** – Reserved for future use.
+- **Nested Bracket Combinations:**  
+  - Examples: `[{}]`, `{{}}`, `<(>)`, `([)])`, `<{>}`  
+    These are reserved to avoid ambiguity.
+- **F-String Brackets (`f"{...}"`)** – For formatted string interpolation.
+
+#### Punctuation Usage
+- Any use of reserved punctuation or bracket combinations outside their valid syntax context (or within unquoted identifiers) will raise a syntax error.
+
+---
+
+### 3.3. String Quotation Rules
+
+Our markup language supports multiple string delimiters to handle both literal and interpolated content:
+
+- **Single Quotes (`'...'`):**  
+  Basic strings with no interpolation.
+  ```toml
+  greeting = 'Hello, World!'
+  ```
+- **Double Quotes (`"..."`):**  
+  Strings that support interpolation.
+  ```toml
+  welcome = "Hello, ${user.name}!"
+  ```
+- **Triple Single Quotes (`'''...'''`):**  
+  Multi-line literal strings.
+  ```toml
+  message = '''This is a
+  multi-line string.'''
+  ```
+- **Triple Double Quotes (`"""..."""`):**  
+  Multi-line strings with interpolation.
+  ```toml
+  announcement = """Version: ${version}
+  Released Today!"""
+  ```
+- **Backticks (`` `...` ``):**  
+  Raw strings where no escaping is performed.
+  ```toml
+  path = `C:\Users\Name`
+  ```
+- **F-Strings (`f"..."`):**  
+  For formatted strings with inline variable interpolation.
+  ```toml
+  configPath = f"${base}/config.toml"
+  ```
+
+#### String Quotation Guidelines
+- Mixing different string delimiters in one literal is not allowed.
+- Unescaped or mismatched quotes within a string will trigger a syntax error.
+
+---
+
+### 3.4. Supported Operators
+
+#### 3.4.1. Arithmetic Operators
+- **Addition:** `+`
+- **Subtraction:** `-`
+- **Multiplication:** `*`
+- **Division:** `/`
+- **Modulo:** `%`
+- **Exponentiation:** `**`
+
+#### 3.4.2. Comparison Operators
+- **Equal to:** `==`
+- **Not equal to:** `!=`
+- **Greater than:** `>`
+- **Less than:** `<`
+- **Greater than or equal to:** `>=`
+- **Less than or equal to:** `<=`
+- **Identity:** `is`
+
+#### 3.4.3. Logical Operators
+- **Logical AND:** `and`
+- **Logical OR:** `or`
+- **Logical NOT:** `not`
+
+#### 3.4.4. Pipeline Operator
+- Chains operations together in a readable sequence:
+  ```toml
+  result = ~( data | transform | filter )
+  ```
+
+#### 3.4.5. Conditional (Ternary) Operator
+- Allows inline conditional expressions:
+  ```toml
+  status = ~( "Active" if is_active else "Inactive" )
+  ```
+
+#### 3.4.6. String Concatenation and Interpolation
+- **Concatenation:** Using the `+` operator.
+- **Interpolation:** Via f-string syntax:
+  ```toml
+  full_path = f"${base}/docs"
+  ```
+
+#### 3.4.7. Membership Operators
+- **Membership:** `in`
+- **Non-membership:** `not in`
+
+#### 3.4.8. Merge Operator
+- Used for table or inline table merging:
+  ```toml
+  [production]
+  << = default_config
+  ```
+
+---
+
+### 3.5. Error Handling
+
+- **Syntax Errors:**  
+  - Reserved keywords or punctuation misused as identifiers will result in a syntax error.
+  - Incorrect use of reserved functions (`File()`, `Git()`) or keywords (such as using `if` or `elif` as variable names) is not permitted.
+- **Bracket Errors:**  
+  - Unmatched, incorrectly nested, or ambiguous bracket combinations will trigger an error.
+- **Invalid Punctuation:**  
+  - Reserved punctuation appearing outside of its valid syntax (or within unquoted keys/identifiers) will raise a clear error.
+- **String Delimiter Errors:**  
+  - Mismatched or unescaped quotes in strings will cause a syntax error.
+
+---
+
+## 4. Examples
+
+### Example 1: Pipeline Operation
+
+```toml
+[processing]
+output = ~( data | sanitize | format )
+```
+
+### Example 2: Valid Use of Reserved Keywords and Conditional Expression
+
+```toml
+[config]
+is_active = true
+status = ~( "Running" if is_active else "Stopped" )
+```
+
+### Example 3: Reserved Function Usage
+
+```toml
+[dependencies]
+source = File("requirements.txt")
+version = Git("https://repo.git", tag="v1.0")
+```
+
+### Example 4: Membership and Merge Operators
+
+```toml
+[settings]
+allowed = ~( "admin" in user.roles )
+merged_config = default << user_override
+```
+
+### Example 5: Invalid Use of Reserved Punctuation
+
+```toml
+[invalid]
+some:key = "value"  # Syntax error: Colon used incorrectly in key name
+```
+
+### Example 6: Invalid Use of Reserved Keyword
+
+```toml
+[config]
+if = "condition"  # Syntax error: 'if' is a reserved keyword
+```
+
+---
+
+## 5. Open Issues
+
+- **Future Reserved Symbols:**  
+  Evaluate whether symbols such as `^`, `&`, or certain bracket combinations should have dedicated functionality.
+- **Keyword Expansion:**  
+  Consider if additional keywords (e.g., `match`) need to be reserved as language features evolve.
+- **Operator Overloading:**  
+  Determine if custom operator definitions for user-defined types are desirable.
+- **Pipeline Syntax Complexity:**  
+  Investigate supporting more complex chaining scenarios or error reporting for pipelines.
+
+---
+
+## 6. Conclusion
+
+MEP-001 clearly defines the reserved keywords, operators, punctuation, and syntax patterns for our markup language. By reserving key elements such as control flow keywords (including `elif`), special functions (`File()`, `Git()`), and specific punctuation and bracket combinations, the specification prevents naming conflicts and ambiguity. Enhanced support for pipelines, conditional expressions, membership tests, and string handling ensures that the language remains both expressive and robust. Feedback is encouraged as we continue refining these rules and explore future language enhancements.