Skip to content

[BUG] post_fp_check_fail does not work for dpdispatcher #737

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
njzjz opened this issue May 25, 2022 · 0 comments · Fixed by #882
Closed

[BUG] post_fp_check_fail does not work for dpdispatcher #737

njzjz opened this issue May 25, 2022 · 0 comments · Fixed by #882
Labels
bug Something isn't working

Comments

@njzjz
Copy link
Member

njzjz commented May 25, 2022

Summary

tag_failure checks the number of the failure tag, but dpdispatcher doesn't have this file.

dpgen/dpgen/generator/run.py

Line 2717 in 0767dce

fp_failed_tags = glob.glob(os.path.join(work_path, 'task.*', 'tag_failure*'))

The alternative way is to check the number of collected data. The failure ratio is $1-\frac{N_\text{data}}{N_\text{task}}$.

DPGEN Version and Platform

devel
@njzjz njzjz added the bug Something isn't working label May 25, 2022
@HuangJiameng HuangJiameng mentioned this issue Aug 17, 2022
Merged
@njzjz njzjz linked a pull request Aug 29, 2022 that will close this issue
fix: check the number of collected data in post_fp_check_fail #882
Merged
AnguseZhang pushed a commit that referenced this issue Sep 1, 2022
@HuangJiameng
Check failed tasks (#882)
dba436a 
see #737 
dpdispatcher use flag_if_job_task_fail to mark the failed jobs, so `post_fp_check_fail` can be used before checking the frames. If we use the alternative way the issue mentioned, we should consider different representative outputs according to `fp_style`. I am wondering if it is a repeat of the following frame checks. However, since `flag_if_job_task_fail` will be marked as True if one task in the group is failed, I am afraid that `rfail` could be high when only a few tasks fail. I'd like to ask for some suggestions.
@njzjz njzjz closed this as completed Sep 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: check the number of collected data in post_fp_check_fail HuangJiameng/dpgen
1 participant
@njzjz