feat: Add materialized helper table in SQLite which unfolds the recursive Segment Group structure + Pydantic/SQLModel and can easily be queried #111

hf-kklein · 2025-04-09T11:37:17Z

Tldr: query ahbs like you'd query PDFs but get tree structure Information for free.

this will be a mid-term replacement for Hochfrequenz/machine-readable_anwendungshandbuecher#336 because it saves us from scraping the .docx files, now that we pay for the XMLs.

wtf

Copilot

Copilot reviewed 5 out of 8 changed files in this pull request and generated 1 comment.

Files not reviewed (3)

domain-specific-terms.txt: Language not supported
src/fundamend/sqlmodels/materialize_ahb_view.sql: Language not supported
tox.ini: Language not supported

Comments suppressed due to low confidence (1)

unittests/test_sqlmodels_anwendungshandbuch.py:83

[nitpick] The variable name 'ahb_is_not_suited_for_equality_comparison' is quite verbose; consider renaming it to something shorter and clearer, for example 'skip_equality_test'.

ahb_is_not_suited_for_equality_comparison = any(x for x in ahb.anwendungsfaelle if x.is_outdated)

src/fundamend/sqlmodels/ahbview.py

Co-authored-by: Copilot <[email protected]>

hf-kklein · 2025-04-09T19:32:00Z

src/fundamend/sqlmodels/anwendungshandbuch.py

@@ -507,3 +507,65 @@ def to_model(self) -> PydanticAnwendungshandbuch:
            pakete=tuple(x.to_model() for x in sorted(self.pakete, key=lambda y: y.position or 0)),
            anwendungsfaelle=tuple(x.to_model() for x in sorted(self.anwendungsfaelle, key=lambda y: y.position or 0)),
        )
+
+


Diese Struktur lässt sich mit wenig Aufwand direkt auf ein flatahb ausm kohlrahbi Mappen.

hf-krechan · 2025-04-10T05:29:08Z

src/fundamend/sqlmodels/anwendungshandbuch.py

hier bietet es sich schon mal an diese Datei aufzuteilen in mehrere Modules.

|- sqlmodels |- anwendungshandbuch |- code.py |- dataelement.py

Aber das ist hier out of scope :)

hf-krechan · 2025-04-10T05:34:06Z

src/fundamend/sqlmodels/anwendungshandbuch.py

+    # Segment Group
+    segmentgroup_id: Optional[str] = Field(default=None, index=True)
+    segmentgroup_name: Optional[str] = Field(default=None, index=True)
+    segmentgroup_ahb_status: Optional[str] = Field(default=None)


ich finde Status beschreibt nicht so gut was dahinter steckt.
Sind hier die Muss [10] bspw. drin oder? Bei AHB Tabellen nennen wir es Pflichtfeld-Kennzeichen.
Auf Englisch vll sowas wie mandatory field indicator

ich würde hier auch die "out of scope"-karte ziehen. Der Spaltenname bildet nur das ab, was genau unter diesem Namen "AHB_Status" aus ursprünglichen einzelnen Tabelle und den XML-dateien kommt:

xml-fundamend-python/unittests/example_files/UTILTS_AHB_1.1d_Konsultationsfassung_2024_04_02.xml

Lines 197 to 204 in 862b448

<S_RFF Name="Referenz auf einen Rechenschritt" Number="00018" AHB_Status="Muss [5]">

<C_C506 Name="Referenz">

<D_1153 Name="Referenz, Qualifier">

<Code Name="Rechenschritt" Description="" AHB_Status="X">Z23</Code>

</D_1153>

<D_1154 Name="Rechenschrittidentifikator" AHB_Status="X [913] [8] ∧ [9]"/>

</C_C506>

</S_RFF>

dort heißt das Ding immer "AHB_Status" und damit man in der gejointen Tabelle sieht, von welcher ebene der jeweilige AHB_Status kam, kommt ein Präfix vorne dran. Mein vorschlag wäre: Wir bauen noch eine weitere Sicht auf die tabelle und ändern da das Naming. Kann ich auch in diesem PR machen, ich würde aber den namen in dieser Tabelle gleich lassen.

okay wenn sie das in den xml auch so nennen passt das für mich :)

weitere sicht ist vertagt

src/fundamend/sqlmodels/ahbview.py

Renames the variable storing the path to the SQL command file to `sql_command_path` for improved clarity and readability. #111 (comment)

README.md

hf-kklein · 2025-04-09T19:34:04Z

src/fundamend/sqlmodels/materialize_ahb_view.sql

+    SELECT
+        h.anwendungsfall_pk,
+        child.primary_key, h.root_id, link.parent_id, h.depth + 1, child.position,
+        h.path || ' > ' || child.name, h.path, h.root_order, 'segment_group',


@hf-krechan statt oder zusätzlich zu den Namen der SG und Segmente könnt man auch ihre keys (SG4 statt Vorgang, DTM statt Datum) nehmen um den Path zu bauen. Dann kann man das "Klapp" Feature bauen

yes please ^^

sieht z.b. so aus jetzt:

id_path': 'SG2>SG5>STS>C_C555>D_4405>Berechnungsformel besitzt keine Rechenoperation>',

src/fundamend/sqlmodels/ahbview.py

README.md

and format sql file... sorry for this

Konstantin added 8 commits April 5, 2025 16:49

wip ahb view

55e7d52

wip

6e538d4

wip

b0c685d

Merge remote-tracking branch 'origin/main' into crate-db

4d97f58

feat: add is_outdated property to Anwendungsfall (interpret ##alt##)

b90ecab

wtf

wip

e036a6c

spellcheck

ab1dd73

remove WIP import

13d48d6

hf-kklein changed the title ~~feat: Add helper view to DB which unrolls recursive structure~~ feat: Add helper view to DB which unrolls the recursive Segment Group structure Apr 9, 2025

hf-kklein changed the title ~~feat: Add helper view to DB which unrolls the recursive Segment Group structure~~ feat: Add helper view to DB which unfolds the recursive Segment Group structure Apr 9, 2025

Konstantin added 6 commits April 9, 2025 14:02

fix snapshot test imports

46fafb3

extend readme

45c9217

Merge remote-tracking branch 'origin/main' into crate-db

8637d2c

extend readme 2

e776980

fix test

99efa66

fix test 2

5f47600

hf-kklein changed the title ~~feat: Add helper view to DB which unfolds the recursive Segment Group structure~~ feat: Add materialized helper view + Pydantic Model which unfolds the recursive Segment Group structure Apr 9, 2025

hf-kklein changed the title ~~feat: Add materialized helper view + Pydantic Model which unfolds the recursive Segment Group structure~~ feat: Add materialized helper view which unfolds the recursive Segment Group structure + Pydantic/SQLModel Apr 9, 2025

hf-kklein requested a review from Copilot April 9, 2025 15:47

Copilot AI reviewed Apr 9, 2025

View reviewed changes

src/fundamend/sqlmodels/ahbview.py Outdated Show resolved Hide resolved

Update src/fundamend/sqlmodels/ahbview.py

f3cf25e

Co-authored-by: Copilot <[email protected]>

hf-kklein marked this pull request as ready for review April 9, 2025 15:48

hf-kklein requested review from lord-haffi, hf-krechan and DeltaDaniel April 9, 2025 15:48

hf-kklein changed the title ~~feat: Add materialized helper view which unfolds the recursive Segment Group structure + Pydantic/SQLModel~~ feat: Add materialized helper view (a plain table in SQLite) which unfolds the recursive Segment Group structure + Pydantic/SQLModel Apr 9, 2025

Konstantin added 2 commits April 9, 2025 18:07

add snapshot test

eed6ae4

Merge remote-tracking branch 'origin/crate-db' into crate-db

2cb3a20

hf-kklein mentioned this pull request Apr 9, 2025

add CI action to create ahb.db from .json files in MR AHB repo Hochfrequenz/machine-readable_anwendungshandbuecher#336

Open

hf-kklein commented Apr 9, 2025

View reviewed changes

hf-krechan reviewed Apr 10, 2025

View reviewed changes

src/fundamend/sqlmodels/ahbview.py Outdated Show resolved Hide resolved

Renames SQL path variable for clarity

065aa3d

Renames the variable storing the path to the SQL command file to `sql_command_path` for improved clarity and readability. #111 (comment)

hf-krechan reviewed Apr 10, 2025

View reviewed changes

README.md Show resolved Hide resolved

hf-krechan approved these changes Apr 10, 2025

View reviewed changes

hf-kklein commented Apr 10, 2025

View reviewed changes

Konstantin and others added 9 commits April 10, 2025 09:39

Merge remote-tracking branch 'origin/main' into crate-db

59e160f

add efoli edifact format version to ahb + view

f3b8cd6

and format sql file... sorry for this

update snapshots

eab5349

fix import orrder

d4776f9

black

48df545

Merge branch 'main' into crate-db

4de6b3d

rework duplicate-check

4500124

add kommunikationvon and beschreibung to helper view

83ad41e

snapshots,

999059c

hf-kklein mentioned this pull request Apr 10, 2025

Create another view that represents the kohlrahbi flatahb #115

Open

Konstantin added 3 commits April 10, 2025 15:47

pylint

eefe528

add id-path

383836d

use code value, not name

3345373

hf-kklein enabled auto-merge (squash) April 10, 2025 14:31

hf-kklein merged commit 4854086 into main Apr 10, 2025
18 checks passed

hf-kklein deleted the crate-db branch April 10, 2025 14:32

	<S_RFF Name="Referenz auf einen Rechenschritt" Number="00018" AHB_Status="Muss [5]">
	<C_C506 Name="Referenz">
	<D_1153 Name="Referenz, Qualifier">
	<Code Name="Rechenschritt" Description="" AHB_Status="X">Z23</Code>
	</D_1153>
	<D_1154 Name="Rechenschrittidentifikator" AHB_Status="X [913] [8] ∧ [9]"/>
	</C_C506>
	</S_RFF>

feat: Add materialized helper table in SQLite which unfolds the recursive Segment Group structure + Pydantic/SQLModel and can easily be queried #111

feat: Add materialized helper table in SQLite which unfolds the recursive Segment Group structure + Pydantic/SQLModel and can easily be queried #111

Uh oh!

Conversation

hf-kklein commented Apr 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hf-kklein commented Apr 9, 2025 •

edited

Loading