Reteti

Reteti is a lexical search experiment based on partitioned index of hashed words in object storage.

Design Objectives

• 1. Fast lexical search with index data based entirely on object storage
• 2. Usability in serverless or scale-to-zero applications for scalability and cost control
• 3. Adaptability to different cloud environments or on-premise systems

Features

• Texts are split to words using normalizers and pre-tokenizers from the Tokenizers Python module.

• A word is any sequence of Unicode lowercase alphanumeric characters between two whitespaces.

• Reteti is language-agnostic and does not use language-specific stemmers.

• Words are hashed and their positions are saved in a partitioned Parquet dataset under hash prefixes.

• Only the Parquet files of the hashed words in a search request are contacted during search.

• Search is performed using DuckDB SQL.

• Words are represented by their hashes or integers during search.

• Original texts are not used to produce a list of matching text IDs.

• Storage and compute are decoupled and Reteti can be used in serverless functions.

• Indexing and searching are separate processes.

• The locations of the index and the texts are independent from one another.

Gradio demo using one million Bulgarian and English short articles is available on Fly.io.
It is scale-to-zero capable and its object storage is managed by Tigris Data.

Search Rules

Search Criteria

Reteti selects the IDs of texts that match the following criteria:

• 1. They have the full set of unique word hashes presented in the search request.
• 2. They have one or more sequences of word hashes identical to the sequence of word hashes in the search request.

Ranking Criterion: Matching Words Frequency

The matching word frequency is the number of search request words found in a document divided by the number of all words in the document. Short documents having high number of matching words are at the top of the search results.

Name

Reteti was one of the giraffe calfs orphaned during a severe drought around 2018 in Northern Kenya and saved thanks to the kindness and efforts of a local community.

Today we use complex data processing technologies thanks to the knowledge, persistence and efforts of many people of a large global community. Just like the small Reteti, we owe much to this community and should always be thankful to its members for their goodwill and contributions!

Thanks and Credits

License

This program is licensed under the terms of the Apache License 2.0.

Author

Dimitar D. Mitov, 2024 - 2025