Skip to content

fix: Ensure problem response reports include all descendant problems regardless of nesting or randomization #36677

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

efortish
Copy link

@efortish efortish commented May 7, 2025

Summary

This PR ensures that the instructor "Problem Responses" report in Open edX includes all student responses to all problems under any selected block, including those that are nested or randomized (such as those from legacy library_content blocks). Previously, the report could miss responses to problems that were not directly visible to the instructor or admin user generating the report, especially in courses using randomized content blocks or deep nesting.

In courses that use randomized content (e.g., legacy library_content blocks) or have deeply nested structures, the instructor dashboard’s problem response report was incomplete. It only included responses to problems visible in the block tree for the user generating the report (typically the admin or instructor). As a result, responses to problems served randomly to students, or problems nested in containers, were omitted from the CSV export. This led to inaccurate reporting and made it difficult for instructors to audit all student answers.


Technical Approach

  • Recursive Expansion:
    The backend now recursively expands any block selected for reporting (not just library_content blocks) to collect all descendant blocks of type problem. This is done regardless of the nesting level or block type.
  • Static/Class Method:
    The logic is encapsulated in a static method (resolve_problem_descendants) within the ProblemResponses class, ensuring clear code organization.
  • Report Generation:
    When generating the report, the backend uses this method to build the list of all relevant problem usage_keys, guaranteeing that all student responses are included in the export, even for randomized or deeply nested problems.
  • Display Name Fallback:
    The code also improves how problem titles are resolved, falling back to the modulestore if the display name is not available in the course block structure.

Impact

  • Instructor Reports:
    Reports now accurately reflect all student responses, regardless of how problems are served or structured in the course.
  • No Student-Facing Changes:
    The change only affects backend report generation; there is no impact on the student experience, grading, or other LMS features.
  • Performance:
    In courses with very large or deeply nested structures, report generation may take slightly longer, but this is necessary to ensure completeness.

How to reproduce:

  • You must import a library with multiple questions(Using legacy content libraries).
  • Use the content library in a unit.
  • In: instructor tab --> data download:

image

  • Select the block that you want to use to generate the report.

  • For this scenario I created 99 users to solve the exam, each user must answer 5 questions, the csv output is supposed to have 495 + 1(labeling row) rows.

  • You will receive much less rows than 496 because the report will only include the responses visible to the user generating the report:

image

  • In this case I received 298 rows, there is a 39.92% of missing data.

Testing

After apply the changes and repeating the process in the how to test section I received:

image

While the data is accurate, showing 496 of 496 expected rows, the "title" column (B) incorrectly displays "problem" across all rows. This happens because the title itself remains hidden if the question is not visible to the user who is generating the report.

That is why I propose the fallback in _build_problem_list, it will allow the CSV task to get the problem title from the modulestorewithout any problem, and the report will looks like:

image

So:

  • Verified that reports generated from the instructor dashboard now include all expected problem responses.
  • Confirmed that randomized problems are present in the CSV export.
  • Checked that the report titles are correctly populated for all problems.

@openedx-webhooks
Copy link

openedx-webhooks commented May 7, 2025

Thanks for the pull request, @efortish!

This repository is currently maintained by @openedx/wg-maintenance-edx-platform.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

  • If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
    • This process (including the steps you'll need to take) is documented here.
  • If it doesn't, simply proceed with the next step.
🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

  • Dependencies

    This PR must be merged before / after / at the same time as ...

  • Blockers

    This PR is waiting for OEP-1234 to be accepted.

  • Timeline information

    This PR must be merged by XX date because ...

  • Partner information

    This is for a course on edx.org.

  • Supporting documentation
  • Relevant Open edX discussion forum threads
🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.


Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

  • The size and impact of the changes that it introduces
  • The need for product review
  • Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

@openedx-webhooks openedx-webhooks added the open-source-contribution PR author is not from Axim or 2U label May 7, 2025
@github-project-automation github-project-automation bot moved this to Needs Triage in Contributions May 7, 2025
@efortish efortish marked this pull request as ready for review May 8, 2025 14:44
@efortish
Copy link
Author

efortish commented May 8, 2025

Hi everyone, I've been working on this solution for generating CSV reports from content_libraries, FYI!
@mariajgrimaldi @MaferMazu @felipemontoya

@mphilbrick211 mphilbrick211 moved this from Needs Triage to Ready for Review in Contributions May 12, 2025
@mphilbrick211 mphilbrick211 added the needs reviewer assigned PR needs to be (re-)assigned a new reviewer label May 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs reviewer assigned PR needs to be (re-)assigned a new reviewer open-source-contribution PR author is not from Axim or 2U
Projects
Status: Ready for Review
Development

Successfully merging this pull request may close these issues.

3 participants