|
1 | 1 | defmodule RuntimeCompiled do
|
| 2 | + @moduledoc """ |
| 3 | + Many ideas from what has been tried at https://github.com/gunnarmorling/1brc/discussions/93 |
| 4 | + Special mentions to @jesperes and @onno-vos-dev for their erlang contributions, I have learned a lot from them. |
| 5 | +
|
| 6 | + Ofc, thanks to @IceDragon200 who started the whole conversation and has been running the benchmarks. |
| 7 | +
|
| 8 | + The main new idea here is to generate a splitter based on the input file at run time, so we |
| 9 | + end up with something like |
| 10 | +
|
| 11 | + def split_lines(<<"city1", ?;, ...>>) ... |
| 12 | + def split_lines(<<"city2", ?;, ...>>) ... |
| 13 | + ... |
| 14 | +
|
| 15 | + This is a very naive implementation, but it's already faster than the previous ones I had. |
| 16 | + I'm sure that there is room for improvement in the parser but I see no point in working on it now. |
| 17 | + At any given time, on my machine the producer process has ~16 messages in the queue, so improvements should probably happen there instead... |
| 18 | +
|
| 19 | +
|
| 20 | + Note: The `Code.compile_string` part is quite cursed, I do not reccomend anyone to actually do this in real life. |
| 21 | + Note2: It was more fun to code than an average macro, though. |
| 22 | + """ |
2 | 23 | @measurement_file "./data/measurements.txt"
|
3 | 24 |
|
4 | 25 | @workers_count :erlang.system_info(:schedulers_online)
|
5 | 26 | @chunk_size 1024 * 1024 * 8
|
6 | 27 |
|
7 | 28 | defmodule CityMatcher do
|
8 | 29 | @moduledoc """
|
9 |
| - This module must be recompiled at runtime, based on the input file. |
| 30 | + This module must be re-compiled at runtime, based on the input file. |
10 | 31 | The functions are generated based on the cities actually found.
|
11 | 32 | """
|
12 | 33 | def process_lines(_), do: raise("Not implemented")
|
|
0 commit comments