Investigate and restart timed-out submissions

[eks-ebi]

Several submissions have been stuck validating and need to be restarted:

✅ 6608c665db8d9f000198b901

✅ 65f1f8330f82060001a1b6fd

🟡 65f9de3adb8d9f0001966a07
-- 🔴 Generate metadata for 65f9de3adb8d9f0001966a07 once the sync is complete.

✅ 66073fe60f82060001a48676

🟡 65d67ba1b73c7400016aa81a
-- 🔴 Generate metadata for 65d67ba1b73c7400016aa81a once the sync is complete.

This has been happening more often recently, so there may be an underlying issue that needs to be investigated.

[karatugo]

✅ 6608c665db8d9f000198b901 - All files are valid, but the status update failed. I have force-pushed to bypass validation; they should be marked as valid shortly.

[karatugo]

✅ 65f1f8330f82060001a1b6fd - The previous failure was due to a bug, now resolved (see EBISPOT/gwas-sumstats-service#308). I have restarted the validation, but encountered a new error: 'template invalid'.

[karatugo]

❌ 65f9de3adb8d9f0001966a07 - It appears to be a network error. I restarted the validation, will follow up on that.

[karatugo]

❌ 66073fe60f82060001a48676 - The studies are across two different folders, causing the validation pipeline to fail in locating them. Is there a policy requiring files to be stored in a single, flat folder? Here are the contents of the specified directory for reference:

ls -ltr <private ftp folder name>
total 64
drwxrws--- 2 gwas_cat spot 1940 Apr 2 23:34 EBIsumstats
drwxrws--- 2 gwas_cat spot 1940 Apr 11 14:38 finalv

Update: ✅ The user fixed the file upload and validation is successful now.

[karatugo]

🟡 65f9de3adb8d9f0001966a07 - Increased mem and time constraints and submitted again for validation.

[karatugo]

🟡 65f9de3adb8d9f0001966a07 - validation passed, waiting for db status update.

For future reference, I unzipped and zipped back the file X10647.18.tsv.gz in the private ftp. The reason was that file description was missing the word 'gzip' as our file extension finder logic depends on that keyword. Details and the ticket for further investigation: EBISPOT/gwas-sumstats-service#317

[Santhi1901]

there is another submission showing VALIDATING: 65d67ba1b73c7400016aa81a. @karatugo, can you please look into this?

[karatugo]

🟡 65f9de3adb8d9f0001966a07 - ran with skipValidation again, hopefully db update won't break this time

Update. DB update worked but the files are not moved to the public ftp.

[sprintell]

This is also linked to this ticket here https://app.zenhub.com/workspaces/gwas-59df823c4a6feb3786810391/issues/gh/ebispot/gwasdepo-deposition-service/161

[karatugo]

🟡 65f9de3adb8d9f0001966a07 - expect the study files in the public ftp tomorrow in a few days. I'll regenerate the metadata files later.

[Santhi1901]

@karatugo

The submission (65f9de3adb8d9f0001966a07) is under embargo, so the sumstats files are not in public FTP.

In production, only some GCSTs have files in the folder.

When I checked, the files were there for GCST90421033-GCST90421797 (I did not check all GCSTs between these, but most of them have sumstats files). GCST90421798-GCST90428040 has empty folders.

[karatugo]

@Santhi1901 I thought we could process all 7,008 files in one sync, but it turns out our system can only process about 700 files each night. It will take several more days to fully sync. If this is too slow, I can explore other solutions, such as initiating a manual sync.

[karatugo]

🟡 65d67ba1b73c7400016aa81a - expect the study files in the public ftp in a few days. I'll regenerate the metadata files later.

[karatugo]

65d67ba1b73c7400016aa81a and 65f9de3adb8d9f0001966a07 are not in the public ftp yet.

[karatugo]

65f9de3adb8d9f0001966a07 is now in the public ftp. Their yamls should be in the public ftp tomorrow (from GCST90421033 to GCST90428040).

[karatugo]

Santhi reported that 65d67ba1b73c7400016aa81a is showing validating again.

[karatugo]

✅ 65d67ba1b73c7400016aa81a is showing submission complete now.

I updated the sumstats meta table as studies were not existing for this submission. Updated the table by a script using PyMongo. For details: see /hps/nobackup/parkinso/spot/gwas/scratch/goci1285

[sprintell]

All the submission has restarted and validated, but some of them are not in the public ftp , ... cannot generate the mssing yaml files, we either wait for them to get to the public ftp, or close them and handle them case by case.

We leave it as it is, new yaml will be generated when curator edits the template