Skip to content

Lineage DAG #252

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 68 commits into from
Feb 18, 2025
Merged
Show file tree
Hide file tree
Changes from 67 commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
2fd7436
Add basic DAG visualization with example job data
techniq Jan 20, 2025
c93932a
Add some top padding to line charts to keep yAxis labels from getting…
techniq Feb 7, 2025
a4fb064
Swap back null ratio charts to use real data (leaving mock value for …
techniq Feb 7, 2025
1f204e6
Remove no longer needed VertxExtensions
techniq Feb 7, 2025
d2caad7
Remove local copy of example job data (now that other PR is merged). …
techniq Feb 10, 2025
d3945db
Fix lint
techniq Feb 10, 2025
ed6fda4
Read sample thrift json join data from filesystem to setup examples
techniq Feb 10, 2025
0554992
fix(FeaturesLineChart): Adjust left padding to fix JSD/Hellinger axis…
techniq Feb 10, 2025
ffef47f
Add `@layerstack/svelte-actions` package and use `remainingViewportHe…
techniq Feb 10, 2025
20b71af
Process `models.source.joinSource.join` to build Lineage DAG
techniq Feb 10, 2025
f1d9a77
Simplify sorting
techniq Feb 11, 2025
5249942
Correct `api.getConf()`type and update specific confs utils to return…
techniq Feb 11, 2025
5bd8288
Include `join` data and allow `joinDrift` to be optional
techniq Feb 11, 2025
7b6ff5c
Remove loading thrift json from the filesystem and allow all join ent…
techniq Feb 11, 2025
2445541
Add/improve empty state messages
techniq Feb 11, 2025
0a2ebeb
Prefer interfaces over concrete types
techniq Feb 11, 2025
e031013
Create `joinToLineage(join)` util to produce `ILineageResponse`, then…
techniq Feb 11, 2025
8ee3e9a
Parallelize `join` and `model` fetchs
techniq Feb 12, 2025
43f2057
Add `api.getJoinLineage({ name })` and use to retrieve transformed jo…
techniq Feb 12, 2025
865a82e
Upgrade LayerChart to 0.97.0 (Apply transformContext for Html layers)
techniq Feb 12, 2025
7b89416
Upgrade LayerChart to 0.97.1 (fix transform dragging when `<Html>` is…
techniq Feb 12, 2025
449729d
Layout nodes using <Html> layer (links still on `<Svg>`) for richer d…
techniq Feb 12, 2025
737eb07
Add basic drilldown dialog (will become drawer with organized info)
techniq Feb 12, 2025
82eafbf
Remove unused exmple data
techniq Feb 12, 2025
a9db9b8
Animate links
techniq Feb 12, 2025
a5c3f71
Upgrade LayerChart to 0.98.0 (zoom/transform to mouse/pointer location)
techniq Feb 13, 2025
cf74733
Upgrade LayerChart to 0.98.1 (account for chart padding when zoom/tra…
techniq Feb 13, 2025
41f586b
Create `nodeEdges` that are relative to each node's left/right midpoi…
techniq Feb 13, 2025
96d8e0f
Upgrade LayerChart to 0.98.2 (fix tool with html layers and mode="man…
techniq Feb 13, 2025
a1c1abf
Upgrade LayerChart to 0.99.0 (Dagre expose `bind:graph`)
techniq Feb 13, 2025
49dea9b
Setup basic tooltip and highight upstream and downstream nodes on hov…
techniq Feb 13, 2025
5251600
Upgrade LayerChart to 0.99.1 (fix html/svg temporarily resetting tran…
techniq Feb 14, 2025
854725b
Refine edge fading with custom function
techniq Feb 14, 2025
4c77309
Add second animated dash abother edge path with color and side based …
techniq Feb 14, 2025
57a00dc
Use common tooltip styling and just pointer offset
techniq Feb 14, 2025
1cca2ed
fix(FeaturesLineChart): Fix legend items fading when selection is made
techniq Feb 14, 2025
f7ae284
Flesh out dialog content
techniq Feb 14, 2025
c27f166
Rename `lineage` to `overview` and make the default tab/route
techniq Feb 14, 2025
738e3e3
Support hiding the tooltip with ⌘/control
techniq Feb 14, 2025
59ba4db
Extract `ConfProperties` component from dialog and show focal entity …
techniq Feb 14, 2025
cd5b33f
Refine types
techniq Feb 14, 2025
e1c40e1
Include `selects` on source tooltip
techniq Feb 15, 2025
8872f24
fix(ConfProperties): Show source with chained join
techniq Feb 15, 2025
ae1cab7
Include `keyColumns` on groupBy tooltip
techniq Feb 15, 2025
461973d
Use Tooltip contained="window" instead of "container". Refine overfl…
techniq Feb 15, 2025
1bb637d
Only show `name` from `metadata` on tooltip to remove noise
techniq Feb 15, 2025
995dfee
feat(ConfProperties): Support showing upstream joinParts and sources …
techniq Feb 15, 2025
6f2e9f6
Transfer connections and infoMap from joinSource join to root join gr…
techniq Feb 15, 2025
1192483
fix(ConfProperties): Pass `includeUpstream` as recursive/self called
techniq Feb 15, 2025
91196cb
fix(ConfProperties): Re-add `name` (useful for drilldown nesting)
techniq Feb 15, 2025
77a6bd6
Use `import Self from './Component.svelte' instead of `<svelte:self>`…
techniq Feb 17, 2025
936d67c
Fix `SOURCE_PROPERTIES` type
techniq Feb 17, 2025
8fcd040
fix(ConfProperties): Fix/workaround remaining svelte-check errors
techniq Feb 17, 2025
fc3c551
fix(ConfProperties): Fix/workaround remaining lint errors
techniq Feb 17, 2025
1df55e7
Fade edge paths on hover
techniq Feb 17, 2025
830039a
Add `svelte-inspect-value` override to fix CI
techniq Feb 17, 2025
5b7c15c
Update svelte-inspect-value version (`0.1.0-next.0` was delisted)
techniq Feb 17, 2025
f669dfb
fix(ConfProperties): Map `operation` to underlying enum name
techniq Feb 17, 2025
fdd1d89
Distinguish between "entity" and "event" source types via icon, and b…
techniq Feb 17, 2025
83168dc
Refine logic to propagate upstream streaming downstream
techniq Feb 17, 2025
c8d347e
Fix check
techniq Feb 17, 2025
e3b0ee7
feat(ConfProperties): Display `join.left` entity/event/joinSource
techniq Feb 18, 2025
4379920
fix(ConfProperties): Exclude `left` when `includeUpstream={false}`
techniq Feb 18, 2025
9fb4bf7
Include `join.left` source as an upstream source connection
techniq Feb 18, 2025
b99cd86
Tweak comment
techniq Feb 18, 2025
77d30d3
Increase top padding to increase chance of showing source => join wit…
techniq Feb 18, 2025
324398e
Job Tracker (#367)
ken-zlai Feb 18, 2025
59f648c
Merge branch 'main' into sean/lineage-dag3
sean-zlai Feb 18, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
115 changes: 94 additions & 21 deletions frontend/package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 6 additions & 1 deletion frontend/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -62,12 +62,14 @@
"dependencies": {
"@creditkarma/thrift-server-core": "^1.0.4",
"@creditkarma/thrift-typescript": "^3.7.6",
"@layerstack/svelte-actions": "^0.0.11",
"@layerstack/utils": "^0.0.7",
"clsx": "^2.1.1",
"d3": "^7.9.0",
"dotenv": "^16.4.7",
"layerchart": "^0.95.0",
"layerchart": "^0.99.1",
"lodash": "^4.17.21",
"svelte-inspect-value": "^0.1.3",
"tailwind-merge": "^2.6.0",
"tailwind-variants": "^0.3.1"
},
Expand All @@ -79,6 +81,9 @@
"nanoid": "^3.3.8",
"svelte-headless-table": {
"svelte": "^5.0.0"
},
"svelte-inspect-value": {
"svelte": "^5.0.0"
}
},
"optionalDependencies": {
Expand Down
60 changes: 42 additions & 18 deletions frontend/src/app.css
Original file line number Diff line number Diff line change
Expand Up @@ -101,25 +101,49 @@

--ring: 0 0% 83.1%;

--job-running-bg: 249 32% 56%;
--job-running-border: 249 54% 72%;
--job-running-active-border: 250 100% 86%;
--job-waiting-bg: 250 30% 24%;
--job-waiting-border: 250 30% 40%;
--job-waiting-active-border: 250 89% 70%;
--job-failed-bg: 10 24% 20%;
--job-failed-border: 10 30% 34%;
--job-failed-active-border: 10 46% 56%;
--job-completed-bg: 160 40% 17%;
--job-completed-border: 160 40% 28%;
--job-completed-active-border: 160 32% 47%;
--job-invalid-bg: 44 80% 14%;
--job-invalid-border: 44 68% 24%;
--job-invalid-active-border: 44 62% 40%;
--job-queued-bg: 0 0% 16%;
--job-queued-border: 0 0% 24%;
--job-queued-active-border: 0 0% 40%;
/* Base colors for reuse */
--color-failed: 10 24% 20%;
--color-failed-border: 10 30% 34%;
--color-failed-active: 10 46% 56%;

--color-running: 250 30% 24%;
--color-running-border: 250 30% 40%;
--color-running-active: 250 48% 62%;

/* Job status colors using base variables */
--job-RUNNING-bg: var(--color-running);
--job-RUNNING-border: var(--color-running-border);
--job-RUNNING-active-border: var(--color-running-active);

--job-WAITING_FOR_UPSTREAM-bg: 210 30% 20%;
--job-WAITING_FOR_UPSTREAM-border: 210 30% 34%;
--job-WAITING_FOR_UPSTREAM-active-border: 210 46% 56%;

--job-FAILED-bg: var(--color-failed);
--job-FAILED-border: var(--color-failed-border);
--job-FAILED-active-border: var(--color-failed-active);

--job-SUCCESS-bg: 160 40% 17%;
--job-SUCCESS-border: 160 40% 28%;
--job-SUCCESS-active-border: 160 32% 47%;

--job-UPSTREAM_FAILED-bg: var(--color-failed);
--job-UPSTREAM_FAILED-border: var(--color-failed-border);
--job-UPSTREAM_FAILED-active-border: var(--color-failed-active);

--job-QUEUED-bg: 0 0% 16%;
--job-QUEUED-border: 0 0% 24%;
--job-QUEUED-active-border: 0 0% 40%;

--job-UPSTREAM_MISSING-bg: 38 30% 20%;
--job-UPSTREAM_MISSING-border: 38 30% 34%;
--job-UPSTREAM_MISSING-active-border: 38 46% 56%;

--job-WAITING_FOR_RESOURCES-bg: var(--color-running);
--job-WAITING_FOR_RESOURCES-border: var(--color-running-border);
--job-WAITING_FOR_RESOURCES-active-border: var(--color-running-active);

/* Scrollbar colors */
--scrollcolor: hsla(0, 0%, 13%, 1);
--scrollbackground: transparent;
scrollbar-color: var(--scrollcolor) var(--scrollbackground);
Expand Down
Loading
Loading