Fix #4659: Restructure doc tree with organized nested directories (#4667)

* Revamp doc tree into nested directories with myst parser

* Segregate existing content for new doc tree structure

* Add faqs and submission process

* Update single quotes to double for formatting

Author: aditi-dsi

docs/source/01-getting-started/README.md

@@ -0,0 +1,8 @@
+# Getting Started
+
+```{toctree}
+:maxdepth: 2
+
+introduction
+installation
+setup/index

docs/source/installation.md renamed to docs/source/01-getting-started/installation.md

@@ -2,7 +2,7 @@
 
 [EvalAI] aims to build a centralized platform to host, participate, and collaborate in Artificial Intelligence (AI) challenges organized around the globe and hope to help in benchmarking progress in AI.
 
-<img src="_static/img/teaser.png">
+<img src="../_static/img/teaser.png">
 
 ## Features
 

docs/source/intro.md renamed to docs/source/01-getting-started/introduction.md

@@ -0,0 +1,3 @@
+# Docker Setup
+
+Guide to setup EvalAI using Docker.

docs/source/01-getting-started/setup/docker-setup.md

@@ -0,0 +1,10 @@
+# Setup
+
+```{toctree}
+:maxdepth: 1
+
+docker-setup
+manual-setup
+linux-setup
+windows-setup
+macos-setup

docs/source/01-getting-started/setup/index.md

@@ -0,0 +1,3 @@
+# Linux Setup
+
+Guide to setup EvalAI on Linux.

docs/source/01-getting-started/setup/linux-setup.md

@@ -0,0 +1,3 @@
+# MacOS Setup
+
+Guide to setup EvalAI on MacOS.

docs/source/01-getting-started/setup/macos-setup.md

@@ -0,0 +1,3 @@
+# Manual Setup
+
+Guide to setup EvalAI manually.

docs/source/01-getting-started/setup/manual-setup.md

@@ -0,0 +1,3 @@
+# Windows Setup
+
+Guide to setup EvalAI on Windows.

docs/source/01-getting-started/setup/windows-setup.md

@@ -0,0 +1,9 @@
+# For Challenge Hosts
+
+```{toctree}
+:maxdepth: 2
+
+hosting-guide/index
+configuration/index
+evaluation/index
+templates/index

docs/source/02-for-challenge-hosts/README.md

@@ -0,0 +1,69 @@
+# Challenge Configuration
+
+Following fields are required (and can be customized) in the [`challenge_config.yml`](https://github.com/Cloud-CV/EvalAI-Starters/blob/master/challenge_config.yaml).
+
+- **title**: Title of the challenge
+
+- **short_description**: Short description of the challenge (preferably 140 characters max)
+
+- **description**: Long description of the challenge (use a relative path for the HTML file, e.g. `templates/description.html`)
+
+- **evaluation_details**: Evaluation details and details of the challenge (use a relative path for the HTML file, e.g. `templates/evaluation_details.html`)
+
+- **terms_and_conditions**: Terms and conditions of the challenge (use a relative path for the HTML file, e.g. `templates/terms_and_conditions.html`)
+
+- **image**: Logo of the challenge (use a relative path for the logo in the zip configuration, e.g. `images/logo/challenge_logo.jpg`). **Note**: The image must be in jpg, jpeg or png format.
+
+- **submission_guidelines**: Submission guidelines of the challenge (use a relative path for the HTML file, e.g. `templates/submission_guidelines.html`)
+
+- **evaluation_script**: Python script which will decide how to evaluate submissions in different phases (path of the evaluation script file or folder relative to this YAML file. For e.g. `evaluation_script/`)
+
+- **remote_evaluation**: True/False (specify whether evaluation will happen on a remote machine or not. Default is `False`)
+
+- **start_date**: Start DateTime of the challenge (Format: YYYY-MM-DD HH:MM:SS, e.g. 2017-07-07 10:10:10) in `UTC` time zone
+
+- **end_date**: End DateTime of the challenge (Format: YYYY-MM-DD HH:MM:SS, e.g. 2017-07-07 10:10:10) in `UTC` time zone
+
+- **published**: True/False (Boolean field that gives the flexibility to publish the challenge once approved by EvalAI admin. Default is `False`)
+
+- **tags**: A list of tags to display the relevant areas of challenge. 
+
+- **domain**: Please choose the relevant domain for your challenge: (CV, NLP, RL, MM, AL, TAB).
+
+- **allowed_email_domains**: A list of domains allowed to participate in the challenge. Leave blank if everyone is allowed to participate. (e.g. `["domain1.com", "domain2.org", "domain3.in"]` Participants with these email domains will only be allowed to participate.)
+
+- **blocked_emails_domains**: A list of domains not allowed to participate in the challenge. Leave blank if everyone is allowed to participate. (e.g. `["domain1.com", "domain2.org", "domain3.in"]` Participants with these email domains will not be allowed to participate.)
+
+- **leaderboard**:
+  A leaderboard for a challenge on EvalAI consists of following subfields:
+
+  - **id**: Unique positive integer field for each leaderboard entry
+
+  - **schema**: Schema field contains the information about the rows of the leaderboard. A schema contains two keys in the leaderboard:
+
+    1. `labels`: Labels are the header rows in the leaderboard according to which the challenge ranking is done.
+
+    2. `default_order_by`: This key decides the default sorting of the leaderboard based on one of the labels defined above.
+    
+    3. `metadata`: This field defines additional information about the metrics that are used to evaluate the challenge submissions.
+
+  The leaderboard schema for the [sample challenge configuration](https://github.com/Cloud-CV/EvalAI-Starters/blob/master/challenge_config.yaml) looks like this:
+
+  ```yaml
+  leaderboard:
+    - id: 1
+      schema:
+        {
+          "labels": ["Metric1", "Metric2", "Metric3", "Total"],
+          "default_order_by": "Total",
+          "metadata": {
+          "Metric1": {
+            "sort_ascending": True,
+            "description": "Metric Description",
+          }
+        }
+  ```
+
+  The above leaderboard schema will look something like this on leaderboard UI:
+
+  ![](../../_static/img/leaderboard.png "Random Number Generator Challenge - Leaderboard")

docs/source/02-for-challenge-hosts/configuration/challenge-config.md

@@ -0,0 +1,38 @@
+# Dataset Splits
+
+Dataset splits define the subset of test-set on which the submissions will be evaluated on. Generally, most challenges have three splits:
+
+  1. **train_split** (Allow participants to make a large number of submissions, let them see how they are doing, and let them overfit)
+  2. **test_split** (Allow a small number of submissions so that they cannot mimic test_set. Use this split to decide the winners for the challenge)
+  3. **val_split** (Allow participants to make submissions and evaluate on the validation split)
+
+  A dataset split has the following subfields:
+
+  - **id**: Unique integer identifier for the split
+
+  - **name**: Name of the split (it must be unique for every split)
+
+  - **codename**: Unique id for each split. Note that the codename of a dataset split is used to map the results returned by the evaluation script to a particular dataset split in EvalAI's database. Please make sure that no two dataset splits have the same codename. Again, make sure that the dataset split's codename match with what is in the evaluation script provided by the challenge host.
+
+- **challenge_phase_splits**:
+
+  A challenge phase split is a relation between a challenge phase and dataset splits for a challenge (many to many relation). This is used to set the privacy of submissions (public/private) to different dataset splits for different challenge phases.
+
+  - **challenge_phase_id**: Id of `challenge_phase` to map with
+
+  - **leaderboard_id**: Id of `leaderboard`
+
+  - **dataset_split_id**: Id of `dataset_split`
+
+  - **visibility**: It will set the visibility of the numbers corresponding to metrics for this `challenge_phase_split`. Select one of the following positive integers based on the visibility level you want: (Optional, Default is `3`)
+
+
+  | Visibility | Description                                                             |
+  | ---------- | ----------------------------------------------------------------------- |
+  | 1          | Only visible to challenge host                                          |
+  | 2          | Only visible to challenge host and participant who made that submission |
+  | 3          | Visible to everyone on leaderboard                                      |
+
+  - **leaderboard_decimal_precision**: A positive integer field used for varying the leaderboard decimal precision. Default value is `2`.
+
+  - **is_leaderboard_order_descending**: True/False (a Boolean field that gives the flexibility to challenge host to change the default leaderboard sorting order. It is useful in cases where you have error as a metric and want to sort the leaderboard in increasing order of error value. Default is `True`)

docs/source/02-for-challenge-hosts/configuration/dataset-splits.md

@@ -0,0 +1,10 @@
+# Configuration
+
+```{toctree}
+:maxdepth: 2
+
+challenge-config
+yaml-reference
+phases-setup
+dataset-splits
+

docs/source/02-for-challenge-hosts/configuration/index.md

@@ -0,0 +1,65 @@
+# Phases Setup
+
+There can be multiple [challenge phases](https://evalai.readthedocs.io/en/latest/glossary.html#challenge-phase) in a challenge. A challenge phase in a challenge contains the following subfields:
+
+  - **id**: Unique integer identifier for the challenge phase
+
+  - **name**: Name of the challenge phase
+
+  - **description**: Long description of the challenge phase (set the relative path of the HTML file, e.g. `templates/challenge_phase_1_description.html`)
+
+  - **leaderboard_public**: True/False (a Boolean field that gives the flexibility to Challenge Hosts to either make the leaderboard public or private. Default is `False`)
+
+  - **is_public**: True/False (a Boolean field that gives the flexibility to Challenge Hosts to either hide or show the challenge phase to participants. Default is `False`)
+
+  - **is_submission_public**: True/False (a Boolean field that gives the flexibility to Challenge Hosts to either make the submissions by default public/private. Note that this will only work when the `leaderboard_public` property is set to true. Default is `False`)
+
+  - **start_date**: Start DateTime of the challenge phase (Format: YYYY-MM-DD HH:MM:SS, e.g. 2017-07-07 10:10:10)
+
+  - **end_date**: End DateTime of the challenge phase (Format: YYYY-MM-DD HH:MM:SS, e.g. 2017-07-07 10:10:10)
+
+  - **test_annotation_file**: This file is used for ranking the submission made by a participant. An annotation file can be shared by more than one challenge phase. (Path of the test annotation file relative to this YAML file, e.g. `annotations/test_annotations_devsplit.json`)
+
+  - **codename**: Unique id for each challenge phase. Note that the codename of a challenge phase is used to map the results returned by the evaluation script to a particular challenge phase. The codename specified here should match with the codename specified in the evaluation script to perfect mapping.
+
+  - **max_submissions_per_day**: A positive integer that tells the maximum number of submissions per day to a challenge phase. (Optional, Default value is `100000`)
+
+  - **max_submissions_per_month**: A positive integer that tells the maximum number of submissions per month to a challenge phase. (Optional, Default value is `100000`)
+
+  - **max_submissions**: A positive integer that decides the maximum number of total submissions that can be made to the challenge phase. (Optional,  Default value is `100000`)
+
+  - **default_submission_meta_attributes**: These are the default metadata attributes that will be displayed for all submissions, the metadata attributes are `method_name`, `method_description`, `project_url`, and `publication_url`.
+      ```yaml
+    default_submission_meta_attributes:
+      - name: method_name
+        is_visible: True
+      - name: method_description
+        is_visible: True
+      - name: project_url
+        is_visible: True
+      - name: publication_url
+        is_visible: True
+      ```
+  - **submission_meta_attributes**: These are the custom metadata attributes that participants can add to their submissions. The custom metadata attributes are `TextAttribute`, `SingleOptionAttribute`, `MultipleChoiceAttribute`, and `TrueFalseField`.
+    ```yaml
+    submission_meta_attributes:
+      - name: TextAttribute
+        description: Sample
+        type: text
+        required: False
+      - name: SingleOptionAttribute
+        description: Sample
+        type: radio
+        options: ["A", "B", "C"]
+      - name: MultipleChoiceAttribute
+        description: Sample
+        type: checkbox
+        options: ["alpha", "beta", "gamma"]
+      - name: TrueFalseField
+        description: Sample
+        type: boolean
+        required: True
+    ```
+  - **is_restricted_to_select_one_submission**: True/False (indicates whether to restrict a user to select only one submission for the leaderboard. Default is `False`)
+  - **is_partial_submission_evaluation_enabled**: True/False (indicates whether partial submission evaluation is enabled. Default is `False`)
+  - **allowed_submission_file_types**: This is a list of file types that are allowed for submission (Optional Default is `.json, .zip, .txt, .tsv, .gz, .csv, .h5, .npy`)

docs/source/02-for-challenge-hosts/configuration/phases-setup.md

@@ -0,0 +1 @@
+# YAML Reference

docs/source/02-for-challenge-hosts/configuration/yaml-reference.md

@@ -0,0 +1,112 @@
+# Evaluation Scripts
+Each challenge has an evaluation script, which evaluates the submission of participants and returns the scores which will populate the leaderboard. The logic for evaluating and judging a submission is customizable and varies from challenge to challenge, but the overall structure of evaluation scripts are fixed due to architectural reasons.
+
+## Writing an Evaluation Script
+
+Evaluation scripts are required to have an `evaluate()` function. This is the main function, which is used by workers to evaluate the submission messages.
+
+The syntax of evaluate function is:
+
+```python
+def evaluate(test_annotation_file, user_annotation_file, phase_codename, **kwargs):
+    pass
+```
+
+It receives three arguments, namely:
+
+- `test_annotation_file`: It represents the local path to the annotation file for the challenge. This is the file uploaded by the Challenge host while creating a challenge.
+
+- `user_annotation_file`: It represents the local path of the file submitted by the user for a particular challenge phase.
+
+- `phase_codename`: It is the `codename` of the challenge phase from the [challenge configuration yaml](https://github.com/Cloud-CV/EvalAI-Starters/blob/master/challenge_config.yaml). This is passed as an argument so that the script can take actions according to the challenge phase.
+
+After reading the files, some custom actions can be performed. This varies per challenge.
+
+The `evaluate()` method also accepts keyword arguments. By default, we provide you metadata of each submission to your challenge which you can use to send notifications to your slack channel or to some other webhook service. Following is an example code showing how to get the submission metadata in your evaluation script and send a slack notification if the accuracy is more than some value `X` (X being 90 in the example given below).
+
+```python
+def evaluate(test_annotation_file, user_annotation_file, phase_codename, **kwargs):
+
+    submission_metadata = kwargs.get("submission_metadata")
+    print submission_metadata
+
+    # Do stuff here
+    # Set `score` to 91 as an example
+
+    score = 91
+    if score > 90:
+        slack_data = kwargs.get("submission_metadata")
+        webhook_url = "Your slack webhook url comes here"
+        # To know more about slack webhook, checkout this link: https://api.slack.com/incoming-webhooks
+
+        response = requests.post(
+            webhook_url,
+            data=json.dumps({'text': "*Flag raised for submission:* \n \n" + str(slack_data)}),
+            headers={'Content-Type': 'application/json'})
+
+    # Do more stuff here
+```
+
+The above example can be modified and used to find if some participant team is cheating or not. There are many more ways for which you can use this metadata.
+
+After all the processing is done, this `evaluate()` should return an output, which is used to populate the leaderboard. The output should be in the following format:
+
+```python
+output = {}
+output['result'] = [
+            {
+                'train_split': {
+                    'Metric1': 123,
+                    'Metric2': 123,
+                    'Metric3': 123,
+                    'Total': 123,
+                }
+            },
+            {
+                'test_split': {
+                    'Metric1': 123,
+                    'Metric2': 123,
+                    'Metric3': 123,
+                    'Total': 123,
+                }
+            }
+        ]
+
+return output
+
+```
+
+Let's break down what is happening in the above code snippet.
+
+1. `output` should contain a key named `result`, which is a list containing entries per dataset split that is available for the challenge phase in consideration (in the function definition of `evaluate()` shown above, the argument: `phase_codename` will receive the _codename_ for the challenge phase against which the submission was made).
+2. Each entry in the list should be a dict that has a key with the corresponding dataset split codename (`train_split` and `test_split` for this example).
+3. Each of these dataset split dict contains various keys (`Metric1`, `Metric2`, `Metric3`, `Total` in this example), which are then displayed as columns in the leaderboard.
+
+## Editing Evaluation Script
+
+Each prediction upload challenge has an evaluation script, which evaluates the submission of participants and returns the scores which will populate the leaderboard. The logic for evaluating a submission is customizable and varies from challenge to challenge.
+
+When setting up a new challenge, hosts need to test multiple versions of evaluation script on EvalAI. To test multiple versions of evaluation script host can update the evaluation script of existing challenge without uploading a whole new challenge configuration.
+
+To edit the evaluation script for existing challenge please follow the following steps:
+
+### 1. Go to the challenge page
+
+Go to hosted challenges and select the challenge to update evaluation script
+
+<img src="../../_static/img/hosted_challenge_page.png"/>
+
+### 2. Navigate to Evaluation criteria tab
+
+Select the Evaluation criteria tab and click on 'upload' button
+
+<img src="../../_static/img/evaluation_criteria_tab.png"/>
+
+### 3. Update the evaluation script
+
+Upload the latest evaluation script and click on 'Submit' button to update the evaluation script
+
+<img src="../../_static/img/edit_evaluation_script.png"/>
+
+**Tada!** you have successfully updated the evaluation script for a challenge. The evaluation workers for the challenge will be restarted automatically to pick up the latest evaluation script. Please wait for a minimum of 10 minutes for the workers to restart.
+

docs/source/02-for-challenge-hosts/evaluation/evaluation-scripts.md

@@ -0,0 +1,10 @@
+# Evaluation
+
+```{toctree}
+:maxdepth: 2
+
+evaluation-scripts
+prediction-upload
+remote-evaluation
+metrics-leaderboards
+

docs/source/02-for-challenge-hosts/evaluation/index.md

@@ -0,0 +1 @@
+# Metric Leaderboards

docs/source/02-for-challenge-hosts/evaluation/metrics-leaderboards.md

@@ -0,0 +1 @@
+# Prediction Upload