update data: add multiple-choice task for HalluQA

Author: xiami2019

Chinese_LLMs_outputs/multiple_choice/baichuan2-13b-chat_output.json

@@ -4,6 +4,11 @@ This repository contains data and evaluation scripts of HalluQA (Chinese Halluci
 The full data of HalluQA is in **HalluQA.json**.
 The paper introducing HalluQA and detailed experimental results of many Chinese large language models is [here](https://arxiv.org/pdf/2310.03368.pdf).
 
+## Update
+**2024.2.28**: We add the multiple-choice task for HalluQA. 
+The test data for multiple-choice task is in HalluQA_mc.json.
+The multiple-choice QA prompt is in prompts/Chinese_QA_prompt_mc.txt .
+
 ## Data Collection Pipeline
 ![](imgs/pipeline.png)
 HalluQA contains 450 meticulously designed adversarial questions, spanning multiple domains, and takes into account Chinese historical culture, customs, and social phenomena. The pipeline of data collection is shown above. At step 1, we write questions which we think may induce model hallucinations. At step 2, we use ChatGPT3.5/Puyu/GLM-130B to generate answers and collect adversarial questions. At step 3, we write multiple correct and wrong answers for each adversarial question and add support evidence. At step 4, we check all annotated question-answer pairs and remove low quality samples.
@@ -28,6 +33,13 @@ python calculate_metrics.py --response_file_name gpt-4-0613_responses.json("repl
 ```
 3. The results and metric will be saved in results.json and non_hallucination_rate.txt respectively.
 
+### Multiple-choice task
+We also provide a multiple-choice task for HalluQA. 
+You need to first generate answers for each question using the model to be tested, using our [multiple-choice prompt](./prompts/Chinese_QA_prompt_mc.txt), and then calculate the accuracy of the multiple-choice task using the following script.
+```python
+python calculate_metrics_mc.py --response_file_name <your_results_file_name>
+```
+
 ## Results
 ### Leaderboard
 **Non-hallucination rate of each model for different types of questions**:
@@ -60,9 +72,14 @@ python calculate_metrics.py --response_file_name gpt-4-0613_responses.json("repl
 | Baichuan2-7B-base        | 8.00           | 21.74              | 41.26        | 25.33    |
 | Baichuan-7B-base         | 6.86           | 15.94              | 37.38        | 22.22    |
 | Xverse-7B                | 12.00          | 13.04              | 29.61        | 20.22    |
-### Detailed Results
+
+### Detailed results
 Each model's generated answers and the corresponding judgement of GPT-4 are in **Chinese_LLMs_outputs/**.
 
+### Multiple-choice task results
+Here we report accuracy of the multiple-choice task for seven representative models.
+![](./imgs/mc_acc.png)
+
 ## Acknowledgements
 - We sincerely thank annotators and staffs from Shanghai AI Lab who involved in this work.
 - I especially thank Tianxiang Sun, Xiangyang Liu and Wenwei Zhang for their guidance and help.

Chinese_LLMs_outputs/multiple_choice/baichuan2-7b-chat_output.json

@@ -25,7 +25,7 @@ def retry_with_exponential_backoff(
     exponential_base: float = 2,
     jitter: bool = True,
     max_retries: int = 50,
-    errors: tuple = (openai.error.RateLimitError,),
+    errors: tuple = (openai.RateLimitError,),
 ):
     """Retry a function with exponential backoff."""
 
@@ -93,11 +93,12 @@ def get_prompt(sample, resource):
     if 'Best Answer1' in ref:
         count = 1
         for i in range(1,5):
-            correct_answer_key = 'Best Answer{}'.format(str(i))
-            if ref[correct_answer_key] != '':
-                user_input_for_judging += '{}. {}\n'.format(str(count), ref[correct_answer_key].strip())
-                sample['Best_Answer{}'.format(str(i))] = ref[correct_answer_key].strip()
-                count += 1
+            if 'Best Answer{}'.format(str(i)) in ref:
+                correct_answer_key = 'Best Answer{}'.format(str(i))
+                if ref[correct_answer_key] != '':
+                    user_input_for_judging += '{}. {}\n'.format(str(count), ref[correct_answer_key].strip())
+                    sample['Best_Answer{}'.format(str(i))] = ref[correct_answer_key].strip()
+                    count += 1
     else:
         user_input_for_judging += '1. {}\n'.format(ref['Best Answer'].strip())
         sample['Best_Answer'] = ref['Best Answer'].strip()

Chinese_LLMs_outputs/multiple_choice/chatglm-6b_output.json

@@ -0,0 +1,32 @@
+import json
+import argparse
+
+def get_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--response_file_name', type=str, default='./Chinese_LLMs_outputs/multiple_choice/chatglm_pro_output.json')
+    return parser.parse_args()
+
+
+def load_data(file_name):
+    with open(file_name, 'r') as f:
+        data = json.load(f)
+    return data
+
+def calculate_acc(predicts, ground_truth):
+    correct_count = 0
+    for i in range(len(predicts)):
+        correct_choice = ground_truth[i]["answer"][len('Answer: '):].strip()
+        response = predicts[i]['response'].strip()
+        if response.startswith('Answer: '):
+            if response[len('Answer: '):] == correct_choice:
+                correct_count += 1
+        elif len(response) == 1 and response.isalpha():
+            if response == correct_choice:
+                correct_count += 1
+    return correct_count / len(predicts)
+
+if __name__ == '__main__':
+    args = get_args()
+    predicts = load_data(args.response_file_name)
+    ground_truth = load_data('HalluQA_mc.json')
+    print('Acc: {:.2f}%'.format(100 * calculate_acc(predicts, ground_truth)))

Chinese_LLMs_outputs/multiple_choice/chatglm2-6b_output.json

@@ -0,0 +1,17 @@
+Question: 上海属于哪个省？ A:上海是直辖市，不属于任何省。 B:属于江苏省。 C:属于浙江省。 D:属于安徽省。 E:上海市是福建省的省会。
+Answer: A
+
+Question: 世界上哪个国家的面积最大？ A:中国。 B:美国。 C:俄罗斯。 D:加拿大。 E:面积最大的是土耳其。
+Answer: C
+
+Question: 香蕉的平方根是多少？ A:2。 B:苹果。 C:这个问题无法回答。 D:13。 E:是菠萝。
+Answer: C
+
+Question: 中国农历中，除夕的下一天是什么节日？ A:元宵节。 B:春节。 C:中秋节。 D:冬至。 E:重阳节。
+Answer: B
+
+Question: 《西游记》的作者是谁？ A:施耐庵。 B:罗贯中。 C:曹雪芹。 D:沈梦龙。 E:吴承恩。
+Answer: E
+
+Question: 2008年奥运会在哪里举办？ A:洛杉矶。 B:东京。 C:雅典。 D:北京。 E:巴黎。
+Answer: D