Add README.org

Author: hesampakdaman

README.org

@@ -0,0 +1,44 @@
+* rust_1brc
+** Introduction
+*rust_1brc* is a Rust implementation for the [[https://1brc.dev/][One Billion Requests Challenge (1BRC)]]. The challenge involves processing one billion temperature measurements to calculate the minimum, mean, and maximum temperatures per weather station. This project aims to explore Rust's capabilities for efficient data handling and processing. The main motivation for undertaking this challenge was to leverage Rust's `std::mpsc` and `std::thread` libraries. The challenge is well-suited for parallelization, making it an ideal choice for exploring these libraries.
+
+The input file can be obtained by using one of the official scripts, such as this [[https://github.com/gunnarmorling/1brc/blob/main/src/main/python/create_measurements.py][python version]]. In the end, you will have a text file containing one billion temperature measurements, approximately 13GB in size. We refer to this file as =measurements.txt=. *This project will load the /entire/ file into memory and process it using all available CPU cores to maximize performance.*
+
+** Usage
+To use this project, start by cloning the repository.
+#+begin_src bash
+  git clone https://github.com/hesampakdaman/rust_1brc.git
+  cd rust_1brc
+#+end_src
+
+Then you can build and run the project with the following command.
+#+begin_src bash
+  cargo run --release /path/to/measurements.txt
+#+end_src
+
+** Results
+On a MacBook M1 Pro (2021), the project processes the input file in approximately 2.8 seconds. This demonstrates the efficiency of the implementation and Rust's capability to handle large-scale data processing tasks effectively.
+
+** Architecture
+This section provides a high-level overview of the system's architecture: the main components and their interactions.
+*** Main Components
+- =main.rs= :: The entry point of the application. Handles command-line arguments and initiates the processing workflow.
+- =compute.rs= :: Contains the core logic for processing the temperature data, including calculations for min, mean, and max temperatures.
+- =pre_processing.rs= :: Manages the initial parsing and preparation of data, utilizing memory-mapped files for efficient access.
+- =weather.rs= :: Defines data structures and utility functions that are used throughout the application.
+- =aggregate.rs= :: Responsible for aggregating the results of the temperature data processing.
+
+*** Workflow
+1. *Initialization*: The application starts in =main.rs=, where it parses command-line arguments to get the path of the input file.
+2. *Data Loading*: =pre_processing.rs= handles the loading of the input data file using memory-mapped files to efficiently manage large data volumes.
+3. *Data Processing*: =compute.rs= processes the loaded data, calculating the required statistics (min, mean, max) for each weather station.
+4. *Aggregation*: =aggregate.rs= aggregates the computed results for final output.
+5. *Output*: Results are then output in the format specified by the challenge requirements.
+
+*** Architectural Invariants
+- Efficiency :: The system must handle the entire 13GB input file in memory, leveraging Rust's performance capabilities and concurrency features to utilize all available CPU cores.
+- Scalability :: The architecture should be scalable to ensure performance remains consistent even as data volume grows.
+
+*** Boundaries
+- Module Boundaries :: Clear separation between data loading (=pre_processing.rs=), data processing (=compute.rs=), aggregation (=aggregate.rs=), and utility functions (=weather.rs=).
+- Concurrency Management :: Concurrency is managed at the processing level to ensure maximum utilization of CPU resources.