Proposal for Ontologies in Sphinx-Needs

[ubmarco]

Introduction

The last Sphinx-Needs user group meetings showed increased interest in the topic of
schema (ontology) validation.

2 previous discussions also address this topic:

• #601 Support for better type/model definition language for Sphinx-Needs
  Started at Jun 9, 2022
• #1087 needs_extra_options only for selected need types
  Started at Dec 20, 2023

This discussion picks up the topic again and proposes a way forward.

Goals of the ontology concept

1. Validate all needs during Sphinx builds.
2. Return meaningful error messages with severity levels.
3. Enhance the editing experience of needs in IDE extensions such as ubCode:
   • Generate schema aware snippets.
   • Schema aware auto-completion.
   • Validate one or multiple needs locally.
   • Validate one or multiple needs with their graph network (links).
     This requires resolution of the whole project incl. dependencies.
4. Support the validation in CLI apps such as ubc, so it can run as a pre-commit hook or in CI pipelines.
5. Come to an understanding of how the ontology interface should look like and
   start a discussion around this.

Note

The Sphinx-Needs internal representation of data is not changing with this proposal.
This may happen in future, as any upstream and downstream tool is interested in the type
and structure of incoming data. Also Sphinx-Needs filters can benefit from typed fields.

Condensed list of known requirements

The following list is a condensed version of the requirements for the technical solution
that were discussed.
The list adhers to RFC 2119 for terms of requirement levels.

1. Schema definition format shall be declarative and agnostic to specific programming languages.
   Rational:
   • No single programming language is used in the ecosystem.
   • Sphinx-Needs is not the only source for ontology definitions in companies.
     It is commonly connected to upstream and downstream data sources that also feature schema definitions.
2. Tool and library support shall be available for Python and Rust.
   Rational:
   • Python is the implementation language of Sphinx-Needs.
   • Rust is the implementation language of ubCode/ubc.
3. Tool and library support shall be available for Linux (x64, arm64), MacOS/Darwin (arm64) and Windows (x64).
4. Mapping of need items to schema types shall be part of the declarative description.
   Rational: Avoid custom code.
5. The solution shall support the definition of default values for extra options.
   Rational: Required for snippet generation and auto-completion.
6. The solution shall work for both core options/links and extra options/links.
7. The solution shall support the required semantics of option and link fields.
8. The solution shall support the conditional required semantics of option and link fields, i.e.
   the existence of a field depends on other field values.
9. The solution shall offer the following need option data types:
   • string default
   • int whole numbers
   • float floating point numbers
   • bool boolean values (understands yes, no, true, false, 1, 0)
   • array of above types; currently only for tags, but can be extended in future to other options as well
10. The solution shall support at least the following string formats:
    • date (ISO 8601)
    • datetime (ISO 8601)
    • time (ISO 8601)
    • duration (ISO 8601)
    • email (RFC 5322)
    • url (RFC 3986)
    • uuid (RFC 4122)
    • enum (string with a list of allowed values)
11. The solution shall support string regex patterns.
12. The solution shall support disallowing additional properties (closing the model).
13. The solution shall support need graph validation, i.e. outgoing need links shall be target to constraints.
    This shall also include chains of links.
    Example: There are 3 need types SPEC, FEAT, USECASE and they link like SPEC > FEAT > USECASE.
    If SPEC has an enum string field asil that can be any of QM|A|B|C|D and, if it is any of A|B|C|D, then
    FEAT must have the field set (required) and it cannot be QM and it must link to a USECASE of the same
    quality.
14. The solution shall be fast to execute for local needs (ms range).
    Rational: Execution as part of a language server.
15. The solution shall be fast to execute for need graphs (seconds range for 20k needs).
    Rational: Reduce waiting time in Sphinx builds and for IDE / CLI apps.
16. The solution should follow an established standard.
    Rational: Avoid reinventing the wheel.
    Examples: JSON Schema, SHACL, Cypher, Protobuf, SysMLv2.
17. The solution should feature an extension or composition mechanism to re-use base definitions
    (think of the types SYS_REQ and SW_REQ that both extend from REQ)
18. The solution may support rendering a visualisation of the schema types and links between them.
19. The solution may calculate or aggregate data during validation.
    Rational: Derive new fields from existing ones.
    Example: Sum all effort float fields of linked needs and validate the result against a threshold.

Note

Many requirements can be fulfilled already by needs_warnings and needs_constraints.
However, complex checks have to be custom coded and also lack performance.
The existing solutions quickly become confusing for bigger projects.

Options

Unfortunately there is no single available solution that ticks all boxes.
The following options have been evaluated:

• JSON Schema

  Idea is to validate the need items directly with JSON schema definitions.
  Validating the full need graph is not possible with JSON schema out of the box.
  This can be done with a custom implementation that uses JSON schema validation.

  • Pros:
    • Declarative & standardized
    • Most developers know it already
    • Mature support in Python and Rust
    • Usable in a language server context
    • Fast
  • Cons:
    • Conditional constraints are not too powerful
    • Error messages can be difficult to understand when using (conditional) schema composition
    • Not made for graph validation

• SHACL

  SHACL is a W3C standard for RDF graphs and can be used to validate the need items.

  • Pros:
    • Declarative & standardized
    • Good support in Python via rdflib and pySHACL
    • Native graph validation
    • Strong modeling capabilities
  • Cons:
    • Not widely known and hard to learn
    • No good Rust support.
      The rudof library lacks important features such as sh:pattern or sh:closed.
      Performance seems good however.
    • Creation of the RDF graph from the need items is required
    • Programmed mapping of need items to SHACL node types is required
    • Understanding the SHACL RDF graph is far from trivial for IDE snippet support

• Graph Query Languages

  The idea is to write graph queries in languages such as Cypher or SPARQL as user input and run them against the need graph in a local graph DB. All items returned are violating the query constraint.

  • Pros:
    • Strong modeling capabilities
    • Graph validation as first class citizen
    • Data aggregations as part of the query
    • Fast execution after the initial graph is built
  • Cons:
    • Hard to understand for users, not standardized in the Sphinx-Needs context
    • Difficult to run in a language server context
    • Hard to update incrementally
    • Not a declarative schema language, hard to read out for IDE snippets
    • Requires a local graph DB (could however be in-process and in-memory, e.g. Neo4j or
      Kuzu).

Solutions

The PR #1441 proposes a solution that is based on JSON schema.
It is a good compromise between the requirements and the available options.
The PR is work-in-progress, however the test cases already lays out how the schema definition would look like:
https://github.com/useblocks/sphinx-needs/tree/mh-schema-validation/tests/doc_test/doc_schema

The approach uses a JSON file that holds a list of schemas.
Each schema may have these keys:

• id a unique identifier for the schema for logging and referencing
• message a user provided error message that is shown when the schema validation fails
• severity the severity of the error, aligned with SHACL to violation, warning and info
• types list of need types the schema applies to; if missing, the schema applies to all need types
• trigger_schema a JSON schema that, if it validates against the need item, activates local_schema and link_schema; this makes it possible to write complex conditional schemas
• local_schema a JSON schema that validates against the unresolved need item, so links will just be
  a list of string need IDs. That is helpful for fast validation because the need graph does not need
  to be built.
• link_schema dictionary that maps link types to a dictionary that constrains resolved need links
  • <link_type> link type for the constraint
    • schema_id the ID of the schema that is used to validate each linked need
    • minItems the minimum number of linked needs that validate against the schema
    • maxItems the maximum number of linked needs that validate against the schema
    • unevaluatedItems whether to complain about additionally linked needs that fail the schema, if the min/maxItems constraint is already met

The link_schema concept also allows modeling chains of links.

Note

Existing SHACL input could be transpiled to this JSON schema solution.

Note

The new schema definition can already be part of exchanged data between projects,
so compatiblity checks become possible.

Going forward

The PR is work-in-progress and needs to be finished (test cases, docs).

I also invite others to comment on the proposal and the PR and also outline validations that are not possible
with the current approach.

I'm also interested in counter arguments and completely different proposals that also tick the most important boxes.

[frank-fzi]

Hi @ubmarco, thanks for starting this discussion! I appreciate that you keep pushing the ontology based validation for Sphinx-Needs. Regarding the Rust support for SHACL: I agree that the rudof library is not yet mature enough and the performance of Python based validation does not scale for large projects. What about Java based alternatives, such as RDF4J or Apache Jena? Both are mature and provide good performance for validation.

[ubmarco]

Hi @frank-fzi thanks for the feedback. Before looking into SHACL, I would like to discuss what validations cannot be done with the proposed idea & implementation.

A general concern I have with SHACL & realtime:
I need to build the RDF graph before running the validation. I think this is a time consuming activity which I do not have currently. For a post-build step the runtime overhead is somewhat ok, but in the language server (LS) we will have performance issues. Incremental updates to the RDF graph are also hard as I have to surgically modify the graph on events (keystrokes, file renaming and others). The updates to the current memory model is already part of the LS backend.

For the libraries, thanks for the suggestions. Fast is always good. Running a Java application on top of Rust in a language server environment feels like the wrong solution from tool perspective.

Let's meet and go into details. I'm curious to find out what constraints cannot be expressed in the proposed solution.
If the problem is not "support for special constraints is missing", I think transpiling SHACL to the JSON schema configuration is a valid approach.

[simon-d-bmw]

Hi,

I've tried out the PR #1441, and I was able to represent our metamodel and its constraints with this JSON schema approach.

For large graph checks, the schemas.json might become complicated, but manageable.
Bottom line: Good solution for introducing ontologies to sphinx-needs.

[AlexanderLanin]

Please note that the solution @simon-d-bmw linked is already in use and represents real world use-cases. We've been pushing eclipse-score/score#751 for quite a while, but should urgently address it. I'll ping @danwos :-)

[rsarrazin2]

Hi @ubmarco,

Thanks for this collection of requirements (I'm missing the Sphinx-Needs version of it, but I understand GitHub doesn't render them [yet]).

In my opinion quite complete. Additionally to requirement 5, the ontology may be used to not only generate snippets, but also to offer e.g. generation of compliant derived needs, generation of compliant links, etc. This may be tricky to realize, though (where to generate the need? with which ID? etc.), so it is at best a may requirement for the backlog.

(Hi @AlexanderLanin, funny to meet good old colleagues in a GitHub issue that isn't C++-related ;-).)