Parquet decoder / decoded page Cache

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

• As described on Experimental parquet decoder with first-class selection pushdown support #6921

There are some times where enabling row filter actually slows down the parquet decoding and one reason for this is having to decompress (e.g. ZSTD) the pages twice

Describe the solution you'd like
#6921 has a cache to avoid this second decode

I would like to get that PR merged

Steps:

• Skip page should also support skip dict page #7409
• Add benchmark for parquet reader with row_filter and project settings #7401
• arrow_reader_row_filter benchmark doesn't capture page cache improvements #7460
• Support sync read for the page cache decode improvement. #7415
• Polish + merge

Describe alternatives you've considered

Additional context

I am filing this as a separate ticket as there are a lot of other ideas on #6921 that make it kind of hard to follow

[alamb]

I have run some performance tests, see:

• WIP: Test enabling Parquet filter pushdown with parquet caching page cache reader datafusion#15506
  I think we will need to look at the ones that report a slowdown more carefully to determine how to regain the performance

[alamb]

Here are the current clickbench results for the code in #6921

• WIP: Test enabling Parquet filter pushdown with parquet caching page cache reader datafusion#15506

--------------------
Benchmark clickbench_1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃  main_base ┃ alamb_filter_pushdown ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     0.57ms │                0.58ms │     no change │
│ QQuery 1     │    68.16ms │               70.64ms │     no change │
│ QQuery 2     │   116.52ms │              118.28ms │     no change │
│ QQuery 3     │   123.86ms │              123.76ms │     no change │
│ QQuery 4     │   776.15ms │              797.72ms │     no change │
│ QQuery 5     │   848.82ms │              880.01ms │     no change │
│ QQuery 6     │    64.80ms │               67.14ms │     no change │
│ QQuery 7     │    77.36ms │               93.04ms │  1.20x slower │
│ QQuery 8     │   957.21ms │              983.79ms │     no change │
│ QQuery 9     │  1239.49ms │             1275.73ms │     no change │
│ QQuery 10    │   299.68ms │              318.01ms │  1.06x slower │
│ QQuery 11    │   344.19ms │              360.58ms │     no change │
│ QQuery 12    │   945.45ms │             1058.68ms │  1.12x slower │
│ QQuery 13    │  1323.58ms │             1548.38ms │  1.17x slower │
│ QQuery 14    │   885.86ms │             1063.57ms │  1.20x slower │
│ QQuery 15    │  1110.56ms │             1134.68ms │     no change │
│ QQuery 16    │  1834.24ms │             1789.95ms │     no change │
│ QQuery 17    │  1662.90ms │             1650.05ms │     no change │
│ QQuery 18    │  3176.45ms │             3164.17ms │     no change │
│ QQuery 19    │   116.43ms │              123.84ms │  1.06x slower │
│ QQuery 20    │  1206.01ms │             1204.47ms │     no change │
│ QQuery 21    │  1445.16ms │             1351.41ms │ +1.07x faster │
│ QQuery 22    │  2708.56ms │             2401.18ms │ +1.13x faster │
│ QQuery 23    │  8690.72ms │             5234.73ms │ +1.66x faster │
│ QQuery 24    │   509.28ms │              684.55ms │  1.34x slower │
│ QQuery 25    │   426.36ms │              553.26ms │  1.30x slower │
│ QQuery 26    │   581.56ms │              802.46ms │  1.38x slower │
│ QQuery 27    │  1797.38ms │             2464.11ms │  1.37x slower │
│ QQuery 28    │ 13274.54ms │            14650.91ms │  1.10x slower │
│ QQuery 29    │   629.26ms │              598.10ms │     no change │
│ QQuery 30    │   970.56ms │             1286.68ms │  1.33x slower │
│ QQuery 31    │  1008.49ms │             1398.40ms │  1.39x slower │
│ QQuery 32    │  3220.54ms │             3249.25ms │     no change │
│ QQuery 33    │  3948.22ms │             3595.57ms │ +1.10x faster │
│ QQuery 34    │  3968.56ms │             3536.51ms │ +1.12x faster │
│ QQuery 35    │  1477.42ms │             1317.28ms │ +1.12x faster │
│ QQuery 36    │   310.58ms │              277.12ms │ +1.12x faster │
│ QQuery 37    │   144.55ms │              136.09ms │ +1.06x faster │
│ QQuery 38    │   188.40ms │              167.13ms │ +1.13x faster │
│ QQuery 39    │   537.25ms │              419.03ms │ +1.28x faster │
│ QQuery 40    │    73.01ms │              110.69ms │  1.52x slower │
│ QQuery 41    │    83.59ms │              105.50ms │  1.26x slower │
│ QQuery 42    │    91.66ms │               96.24ms │     no change │
└──────────────┴────────────┴───────────────────────┴───────────────┘

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                    ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (main_base)               │ 63263.96ms │
│ Total Time (alamb_filter_pushdown)   │ 62263.28ms │
│ Average Time (main_base)             │  1471.25ms │
│ Average Time (alamb_filter_pushdown) │  1447.98ms │
│ Queries Faster                       │         10 │
│ Queries Slower                       │         15 │
│ Queries with No Change               │         18 │
└──────────────────────────────────────┴────────────┘

[alamb]

I did some analysis on the queries that show the largest slowdown:

│ QQuery 24    │   509.28ms │              684.55ms │  1.34x slower │
│ QQuery 25    │   426.36ms │              553.26ms │  1.30x slower │
│ QQuery 26    │   581.56ms │              802.46ms │  1.38x slower │
...
│ QQuery 30    │   970.56ms │             1286.68ms │  1.33x slower │
│ QQuery 31    │  1008.49ms │             1398.40ms │  1.39x slower │

Query 24-26
sql
SELECT "SearchPhrase" FROM hits WHERE "SearchPhrase" <> '' ORDER BY to_timestamp_seconds("EventTime") LIMIT 10;
SELECT "SearchPhrase" FROM hits WHERE "SearchPhrase" <> '' ORDER BY "SearchPhrase" LIMIT 10;
SELECT "SearchPhrase" FROM hits WHERE "SearchPhrase" <> '' ORDER BY to_timestamp_seconds("EventTime"), "SearchPhrase" LIMIT 10;

Queries 30-31

SELECT "SearchEngineID", "ClientIP", COUNT(*) AS c, SUM("IsRefresh"), AVG("ResolutionWidth") FROM hits WHERE "SearchPhrase" <> '' GROUP BY "SearchEngineID", "ClientIP" ORDER BY c DESC LIMIT 10;
SELECT "WatchID", "ClientIP", COUNT(*) AS c, SUM("IsRefresh"), AVG("ResolutionWidth") FROM hits WHERE "SearchPhrase" <> '' GROUP BY "WatchID", "ClientIP" ORDER BY c DESC LIMIT 10;

Basically they all have the filter SearchPhrase <> '' (cleaning out empty strings)

How selective is SearchPhrase <> ''?

This predicate select 13/99M

> select count(*) from 'hits.parquet' WHERE "SearchPhrase" <> '';
+----------+
| count(*) |
+----------+
| 13172392 |
+----------+
1 row(s) fetched.
Elapsed 0.303 seconds.

> select count(*) from 'hits.parquet';
+----------+
| count(*) |
+----------+
| 99997497 |
+----------+
1 row(s) fetched.

I am now profiling to determine the root cause

[alamb]

I profiled this command:

./datafusion-cli-filter-pushdown -c "SELECT \"WatchID\", \"ClientIP\", COUNT(*) AS c, SUM(\"IsRefresh\"), AVG(\"ResolutionWidth\") FROM hits WHERE \"SearchPhrase\" <> '' GROUP BY \"WatchID\", \"ClientIP\" ORDER BY c DESC LIMIT 10;"

And I see that about 15% of the time is being spent "skipping" records -- aka evaluating the RowSelection in the parquet reader

This is likely the same observation that lead @XiangpengHao to propose adding bitmask support in

• [Parquet] Add BooleanArray based row selection #6624

I need to think about what the right next steps are

[XiangpengHao]

This is likely the same observation that lead @XiangpengHao to propose adding bitmask support in

Exactly, the predicate SearchPhase <> '' is not selective, meaning that we have lots of small selections:

select 2 
skip 3
select 4
....

Each selector is 16 bytes, causing a lot of memory overhead.

This also means that for each small select and skip, we will call the corresponding read_records and skip_records. These methods are currently not super optimized for millions of calls.

The associated and_then method is also expensive to evaluate.

[XiangpengHao]

I need to think about what the right next steps are

One potential item I can help is to make the reader a optional opt-in reader, and merge it in.

With that, we will have a testbed for future optimization like better selection mechanisms.

[alamb]

I need to think about what the right next steps are

One potential item I can help is to make the reader a optional opt-in reader, and merge it in.

With that, we will have a testbed for future optimization like better selection mechanisms.

I think this is a good idea

[alamb]

Actually, I think all we need to show is that the code in arrow-rs helps in some cases and doesn't degrade performance for others -- we don't need to show that it completely fixes the issue downstream in DataFusion

I will try and gather evidence to determine if the decoded page cache can be always be used

[zhuqi-lucas]

I am interested for this topic, if anything i can help the testing to compare the performance or code improvement as a follow-up?