Skip to content

Commit e27ed42

Browse files
keirpkeirp
committed
Initial commit
0 parents  commit e27ed42

34 files changed

+55139
-0
lines changed

‎.DS_Store

6 KB
Binary file not shown.

‎README.md

+114
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
# <img align="left" width="50" height="50" src="imgs/openwebmath_logo.png" style="margin-top: -12px; margin-bottom: 0px;"> OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text
2+
[Keiran Paster](https://keirp.com)\*, [Marco Dos Santos](#)\*, [Zhangir Azerbayev](https://zhangir-azerbayev.github.io/), [Jimmy Ba](https://jimmylba.github.io/)
3+
4+
5+
This codebase contains the code for the following stages of the OpenWebMath pipeline:
6+
7+
1. **Prefiltering**: Fast filters to remove most of the non-mathematical web documents from Common Crawl.
8+
2. **Text Extraction**: Extracting text and LaTeX from HTML documents.
9+
3. **Language Identification**: Identifying the language of the extracted text and filtering out non-English documents.
10+
4. **MathScore Filtering**: Filtering out documents with low *MathScores*.
11+
5. **Perplexity Filtering**: Filtering out documents with high perplexity.
12+
6. **Deduplication**: Removing duplicate documents.
13+
14+
**OpenWebMath** is a dataset containing the majority of the high-quality, mathematical text from the internet. It is filtered and extracted from over 200B HTML files on Common Crawl down to a set of **6.3 million documents** containing a total of **14.7B tokens**. OpenWebMath is intended for use in *pretraining* and *finetuning* large language models.
15+
16+
You can download the dataset using Hugging Face:
17+
18+
```python
19+
from datasets import load_dataset
20+
ds = load_dataset("open-web-math/open-web-math")
21+
```
22+
23+
Important Links:
24+
- [ArXiv Paper](...)
25+
- [GitHub Repository](...)
26+
- [Paper Twitter Thread](...)
27+
28+
# OpenWebMath Contents
29+
30+
The dataset is structured as follows:
31+
32+
```python
33+
{
34+
"text": ..., # document text.
35+
"url": ..., # document url.
36+
"date": ..., # date the page was crawled.
37+
"metadata": ..., # JSON containing information from the extraction process.
38+
}
39+
```
40+
41+
OpenWebMath contains documents from over 130k different domains, including data from forums, educational pages, and blogs. The dataset contains documents covering mathematics, physics, statistics, computer science, and more. The following table shows the most common domains in OpenWebMath by character count.
42+
43+
| Domain | # Characters | % Characters |
44+
|-----------------------|--------------|--------------|
45+
| stackexchange.com | 4,655,132,784| 9.55% |
46+
| nature.com | 1,529,935,838| 3.14% |
47+
| wordpress.com | 1,294,166,938| 2.66% |
48+
| physicsforums.com | 1,160,137,919| 2.38% |
49+
| github.io | 725,689,722 | 1.49% |
50+
| zbmath.org | 620,019,503 | 1.27% |
51+
| wikipedia.org | 618,024,754 | 1.27% |
52+
| groundai.com | 545,214,990 | 1.12% |
53+
| blogspot.com | 520,392,333 | 1.07% |
54+
| mathoverflow.net | 499,102,560 | 1.02% |
55+
56+
# OpenWebMath Pipeline
57+
58+
<img src="imgs/pipeline.png" alt="Overview of the OpenWebMath Pipeline">
59+
60+
OpenWebMath builds on the massive [Common Crawl](https://commoncrawl.org/) dataset, which contains over 200B HTML documents. We filtered the data to only include documents that are: (1) in English, (2) contain mathematical content, and (3) are of high quality. We also put a strong emphasis on extracting LaTeX content from the HTML documents as well as reducing boilerplate in comparison to other web datasets.
61+
62+
The OpenWebMath pipeline consists of five steps:
63+
1. **Prefiltering HTML Documents**:
64+
- We apply a simple prefilter to all HTML documents in Common Crawl in order to skip documents without mathematical content to unnecessary processing time.
65+
66+
2. **Text Extraction**:
67+
- Extract text, including LaTeX content, from the HTML documents while removing boilerplate.
68+
69+
3. **Content Classification and Filtering**:
70+
- Apply a [FastText language identification model](https://fasttext.cc/docs/en/language-identification.html) to keep only English documents.
71+
- Filter high perplexity documents using a [KenLM](https://github.com/kpu/kenlm) model trained on [Proof-Pile](https://huggingface.co/datasets/hoskinson-center/proof-pile).
72+
- Filter non-mathematical documents using our own *MathScore* model.
73+
74+
4. **Deduplication**:
75+
- Deduplicate the dataset using SimHash in [text-dedup](https://github.com/ChenghaoMou/text-dedup).
76+
77+
5. **Manual Inspection**:
78+
- Inspect the documents gathered from previous steps and remove low quality pages.
79+
80+
For a detailed discussion on the processing pipeline, please refer to our paper.
81+
82+
# License
83+
84+
OpenWebMath is made available under an ODC-By 1.0 license; users should also abide by the CommonCrawl ToU: [https://commoncrawl.org/terms-of-use/](https://commoncrawl.org/terms-of-use/). We do not alter the license of any of the underlying data.
85+
86+
# Citation Information
87+
88+
```
89+
@article{openwebmath,
90+
title={OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text},
91+
author={Keiran Paster and Marco Dos Santos and Zhangir Azerbayev and Jimmy Ba},
92+
journal={arXiv preprint arXiv:????.?????},
93+
eprint={????.?????},
94+
eprinttype = {arXiv},
95+
url={https://arxiv.org/abs/????.?????},
96+
year={2023}
97+
}
98+
```
99+
100+
# Code Structure
101+
102+
We provide code in this repository to reproduce our processing pipeline. The code is organized into three separate folders:
103+
104+
1. `text_extraction` contains the code for extracting text and LaTeX from HTML documents.
105+
2. `extract_from_cc` contains the code for extracting the dataset from Common Crawl, including prefiltering, language identification, MathScore filtering, and perplexity filtering.
106+
3. `filtering` includes many of the manual filtering steps, including blacklisted domains.
107+
108+
In order to run the `extract_from_cc` code, you either need to run it in Apache Spark or manually run `extract_from_warc.py` by passing in a WARC file as an argument.
109+
110+
For deduplication, please use the [text-dedup](https://github.com/ChenghaoMou/text-dedup) library.
111+
112+
Finally, for filtering, `filter.py` contains the code to load a Hugging Face dataset and filter it based on our heuristics.
113+
114+
The *MathScore* model and KenLM model will be released in the near future.

‎extract_from_cc/.DS_Store

6 KB
Binary file not shown.

‎extract_from_cc/configs/randomized_all.yaml

+35
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
markdown_headings: [
2+
[0.9, True],
3+
[0.1, False],
4+
]
5+
markdown_code: [
6+
[0.95, True],
7+
[0.05, False],
8+
]
9+
boilerplate_config:
10+
ratio_threshold: [
11+
[0.9, 0.18],
12+
[0.1, 0.30],
13+
]
14+
absolute_threshold: [
15+
[0.9, 10],
16+
[0.1, 20],
17+
]
18+
end_threshold: [
19+
[0.95, 15],
20+
[0.05, 5],
21+
]
22+
enable: [
23+
[0.95, True],
24+
[0.05, False],
25+
]
26+
remove_buttons: True
27+
remove_image_figures: True
28+
remove_link_clusters: True
29+
table_config:
30+
min_rows: 2
31+
min_cols: 3
32+
format: 'plain'
33+
remove_chinese: True
34+
remove_edit_buttons: True
35+
extract_latex: True

0 commit comments

Comments
 (0)