Optimise Storage of Validation Results in MongoDB

[karatugo]

Description:

The current implementation of store_validation_results_in_db is inefficient as it performs multiple database updates for every field (retrieved, data_valid, and error_code) for every study, resulting in 3 x number of studies queries to MongoDB. This inefficiency impacts performance, particularly when processing a large number of studies.

Proposed Solutions:

• Batch Update Based on Callback ID:

  • Group all validation results by callbackID and perform a single bulk update in MongoDB.
  • This will reduce the number of database interactions significantly by updating all records related to the same callbackID in one query.

• Selective Update for Failed Cases:

  • Detect studies that failed (error_code != None) and update them separately.
  • Perform a bulk update for studies without errors, avoiding the overhead of individual field updates.

• Consolidate Field Updates:

  • Instead of calling store_retrieved_status(), store_data_valid_status(), and store_error_code() individually for each study, combine these into a single MongoDB document update operation.