Add benchmark for measuring create_block_mask creation time

[drisspg]

Stack from ghstack (oldest at bottom):

• -> Add benchmark for measuring create_block_mask creation time #137

Pull-Request-resolved: #136

Need to figure out why memory allocated is not behaving correctly

❯ python benchmarks/bench_block_mask.py
Running 15 benchmark configurations...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:14<00:00,  1.03it/s]
+-----+-----+-------+-------+------------------------+----------------------+----------------+
|   B |   H |     M |     N | Mask Mod               |   Creation Time (ms) |   Memory (GiB) |
+=====+=====+=======+=======+========================+======================+================+
|   1 |   8 |  8192 |  8192 | causal                 |               0.1657 |              0 |
+-----+-----+-------+-------+------------------------+----------------------+----------------+
|   1 |   8 |  8192 |  8192 | sliding_window         |               0.226  |              0 |
+-----+-----+-------+-------+------------------------+----------------------+----------------+
|   1 |   8 |  8192 |  8192 | prefix_lm              |               0.1693 |              0 |
+-----+-----+-------+-------+------------------------+----------------------+----------------+
|   1 |   8 |  8192 |  8192 | doc_mask_mod           |               1.0622 |              0 |
+-----+-----+-------+-------+------------------------+----------------------+----------------+
|   1 |   8 |  8192 |  8192 | dilated_sliding_window |               0.4233 |              0 |
+-----+-----+-------+-------+------------------------+----------------------+----------------+
|   1 |   8 | 16384 | 16384 | causal                 |               1.5539 |              0 |
+-----+-----+-------+-------+------------------------+----------------------+----------------+
|   1 |   8 | 16384 | 16384 | sliding_window         |               2.8478 |              0 |
+-----+-----+-------+-------+------------------------+----------------------+----------------+
|   1 |   8 | 16384 | 16384 | prefix_lm              |               1.598  |              0 |
+-----+-----+-------+-------+------------------------+----------------------+----------------+
|   1 |   8 | 16384 | 16384 | doc_mask_mod           |               7.6279 |              0 |
+-----+-----+-------+-------+------------------------+----------------------+----------------+
|   1 |   8 | 16384 | 16384 | dilated_sliding_window |               4.9444 |              0 |
+-----+-----+-------+-------+------------------------+----------------------+----------------+
|   1 |   8 | 32768 | 32768 | causal                 |               6.0614 |              0 |
+-----+-----+-------+-------+------------------------+----------------------+----------------+
|   1 |   8 | 32768 | 32768 | sliding_window         |              11.2369 |              0 |
+-----+-----+-------+-------+------------------------+----------------------+----------------+
|   1 |   8 | 32768 | 32768 | prefix_lm              |               6.2164 |              0 |
+-----+-----+-------+-------+------------------------+----------------------+----------------+
|   1 |   8 | 32768 | 32768 | doc_mask_mod           |              30.3449 |              0 |
+-----+-----+-------+-------+------------------------+----------------------+----------------+
|   1 |   8 | 32768 | 32768 | dilated_sliding_window |              19.6154 |              0 |
+-----+-----+-------+-------+------------------------+----------------------+----------------+