Athena: Add a llm as profiler approach for text exercise feedback generation
#105
Conversation
|
There hasn't been any activity on this pull request recently. Therefore, this pull request has been automatically marked as stale and will be closed if no further activity occurs within seven days. Thank you for your contributions. |
|
There hasn't been any activity on this pull request recently. Therefore, this pull request has been automatically marked as stale and will be closed if no further activity occurs within seven days. Thank you for your contributions. |
WalkthroughA new "LLM as Profiler" feature is introduced for text-based exercises, including configuration, prompt templates, and data models for competency profiling and feedback suggestion. This addition enables AI-driven analysis of student submissions, mapping competencies, evaluating evidence, and generating structured, actionable feedback aligned with grading rubrics. Changes
Sequence Diagram(s)sequenceDiagram
participant Teacher
participant LLMAsProfilerConfig
participant ProfilerPrompt
participant AI_Model
participant GenerateSuggestionsPrompt
Teacher->>LLMAsProfilerConfig: generate_suggestions(exercise, submission, ...)
LLMAsProfilerConfig->>ProfilerPrompt: Prepare competency profiling prompt
ProfilerPrompt->>AI_Model: Analyze submission for competencies
AI_Model-->>ProfilerPrompt: SubmissionCompetencyProfile
ProfilerPrompt->>GenerateSuggestionsPrompt: Prepare feedback prompt with profile
GenerateSuggestionsPrompt->>AI_Model: Generate actionable feedback
AI_Model-->>GenerateSuggestionsPrompt: AssessmentModel (feedback list)
GenerateSuggestionsPrompt-->>LLMAsProfilerConfig: Parsed feedback suggestions
LLMAsProfilerConfig-->>Teacher: Return feedback suggestions
Possibly related issues
Suggested labels
Poem
📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (2)
🧰 Additional context used🧠 Learnings (2)📓 Common learningsathena/modules/text/module_text_llm/module_text_llm/llm_as_profiler/generate_suggestions.py (2)🧬 Code Graph Analysis (1)athena/modules/text/module_text_llm/module_text_llm/llm_as_profiler/generate_suggestions.py (5)
🪛 Pylint (3.3.7)athena/modules/text/module_text_llm/module_text_llm/llm_as_profiler/prompt_generate_feedback.py[convention] 86-86: Line too long (116/100) (C0301) [convention] 88-88: Line too long (114/100) (C0301) [convention] 93-93: Line too long (107/100) (C0301) [convention] 94-94: Line too long (118/100) (C0301) [convention] 95-95: Line too long (108/100) (C0301) [convention] 100-100: Line too long (130/100) (C0301) [convention] 103-103: Line too long (124/100) (C0301) [convention] 109-109: Line too long (109/100) (C0301) [convention] 112-112: Line too long (107/100) (C0301) [convention] 119-119: Line too long (111/100) (C0301) [convention] 1-1: Missing module docstring (C0114) [error] 1-1: Unable to import 'pydantic' (E0401) [convention] 5-5: Constant name "system_message" doesn't conform to UPPER_CASE naming style (C0103) [convention] 68-68: Constant name "human_message" doesn't conform to UPPER_CASE naming style (C0103) [refactor] 78-78: Too few public methods (0/2) (R0903) [convention] 92-92: Missing class docstring (C0115) [convention] 98-98: Missing class docstring (C0115) [refactor] 98-98: Too few public methods (0/2) (R0903) [refactor] 123-123: Too few public methods (0/2) (R0903) [convention] 2-2: standard import "enum.Enum" should be placed before third party import "pydantic.BaseModel" (C0411) [convention] 3-3: standard import "typing.List" should be placed before third party import "pydantic.BaseModel" (C0411) athena/modules/text/module_text_llm/module_text_llm/llm_as_profiler/generate_suggestions.py[convention] 18-18: Line too long (111/100) (C0301) [convention] 23-23: Line too long (118/100) (C0301) [convention] 52-52: Line too long (120/100) (C0301) [convention] 71-71: Line too long (118/100) (C0301) [convention] 111-111: Line too long (104/100) (C0301) [convention] 113-113: Line too long (136/100) (C0301) [convention] 1-1: Missing module docstring (C0114) [error] 3-3: Unable to import 'athena' (E0401) [error] 4-4: Unable to import 'athena.text' (E0401) [error] 5-5: Unable to import 'athena.logger' (E0401) [error] 6-10: Unable to import 'llm_core.utils.llm_utils' (E0401) [error] 11-11: Unable to import 'llm_core.utils.predict_and_parse' (E0401) [convention] 18-18: Missing function or method docstring (C0116) [refactor] 18-18: Too many local variables (20/15) (R0914) [convention] 6-10: third party import "llm_core.utils.llm_utils.get_chat_prompt_with_formatting_instructions" should be placed before first party imports "athena.emit_meta", "athena.text.Exercise", "athena.logger.logger" (C0411) [convention] 11-11: third party import "llm_core.utils.predict_and_parse.predict_and_parse" should be placed before first party imports "athena.emit_meta", "athena.text.Exercise", "athena.logger.logger" (C0411) [convention] 12-12: third party import "module_text_llm.approach_config.ApproachConfig" should be placed before first party imports "athena.emit_meta", "athena.text.Exercise", "athena.logger.logger" (C0411) [convention] 13-14: third party import "module_text_llm.helpers.utils.add_sentence_numbers" should be placed before first party imports "athena.emit_meta", "athena.text.Exercise", "athena.logger.logger" (C0411) [convention] 15-15: third party import "module_text_llm.llm_as_profiler.prompt_profiler.SubmissionCompetencyProfile" should be placed before first party imports "athena.emit_meta", "athena.text.Exercise", "athena.logger.logger" (C0411) [convention] 16-16: third party import "module_text_llm.llm_as_profiler.prompt_generate_feedback.AssessmentModel" should be placed before first party imports "athena.emit_meta", "athena.text.Exercise", "athena.logger.logger" (C0411) ⏰ Context from checks skipped due to timeout of 90000ms (3)
🔇 Additional comments (4)
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 7
🧹 Nitpick comments (5)
athena/modules/text/module_text_llm/module_text_llm/llm_as_profiler/prompt_generate_feedback.py (2)
5-65: Review the prompt template for clarity and consistency.The system message is comprehensive and well-structured. However, there are a few areas that could be improved:
- Line 30: The instruction "Tell them 'Go over this subject/topic'" is somewhat vague
- Lines 41-47: The guidelines section has good coverage but could benefit from consistent formatting
- Line 69: There's an escaped quote that might cause formatting issues
Consider this refinement for better clarity:
- - Review Concept: When student faces conceptual misunderstandings; suggest them to revisit foundational material. Tell them "Go over this subject/topic" without explaining/revealing answer. + - Review Concept: When student faces conceptual misunderstandings; suggest them to revisit foundational material. Provide specific topics to review without explaining/revealing the answer.And fix the escaped quote:
-Student\'s submission to grade (with sentence numbers <number>: <sentence>): +Student's submission to grade (with sentence numbers <number>: <sentence>):
96-118: Consider validation for line number consistency.The
FeedbackModelis well-structured, but there's no validation to ensureline_start <= line_endwhen both are provided.Add a validator to ensure line number consistency:
from pydantic import BaseModel, Field, validator class FeedbackModel(BaseModel): # ... existing fields ... + @validator('line_end') + def validate_line_range(cls, v, values): + if v is not None and values.get('line_start') is not None: + if v < values['line_start']: + raise ValueError('line_end must be greater than or equal to line_start') + return vathena/modules/text/module_text_llm/module_text_llm/llm_as_profiler/prompt_profiler.py (3)
42-47: Fix the escaped quote for consistency.Similar to the other file, there's an escaped quote that should be corrected.
-Student\'s submission to grade (with sentence numbers <number>: <sentence>): +Student's submission to grade (with sentence numbers <number>: <sentence>):
51-60: Clarify the docstring and available features.The docstring lists many features that may not all be relevant or available in this specific prompt context.
Consider simplifying the docstring to reflect actual usage:
class ProfilerPrompt(BaseModel): - """\ -Features available: **{problem_statement}**, **{example_solution}**, **{grading_instructions}**, **{max_points}**, **{bonus_points}**, **{submission}**, **{practical_theoretical}**, **{creative_guidance}**, **{followup_summary}**, **{brief_detailed}** - -_Note: **{problem_statement}**, **{example_solution}**, or **{grading_instructions}** might be omitted if the input is too long._\ -""" + """ + Prompt structure for competency profiling. + + Template variables: problem_statement, example_solution, grading_instructions, submission. + Note: Some variables might be omitted if input is too long. + """
95-109: Consider validation for line number consistency.Similar to the feedback model, this could benefit from line range validation.
Add the same line validation as suggested for the feedback model:
from pydantic import BaseModel, Field, validator class CompetencyEvaluation(BaseModel): # ... existing fields ... + @validator('line_end') + def validate_line_range(cls, v, values): + if v is not None and values.get('line_start') is not None: + if v < values['line_start']: + raise ValueError('line_end must be greater than or equal to line_start') + return v
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (5)
athena/modules/text/module_text_llm/module_text_llm/config.py(1 hunks)athena/modules/text/module_text_llm/module_text_llm/llm_as_profiler/__init__.py(1 hunks)athena/modules/text/module_text_llm/module_text_llm/llm_as_profiler/generate_suggestions.py(1 hunks)athena/modules/text/module_text_llm/module_text_llm/llm_as_profiler/prompt_generate_feedback.py(1 hunks)athena/modules/text/module_text_llm/module_text_llm/llm_as_profiler/prompt_profiler.py(1 hunks)
🧰 Additional context used
🧬 Code Graph Analysis (1)
athena/modules/text/module_text_llm/module_text_llm/llm_as_profiler/__init__.py (5)
athena/modules/text/module_text_llm/module_text_llm/approach_config.py (1)
ApproachConfig(6-16)athena/modules/text/module_text_llm/module_text_llm/llm_as_profiler/prompt_generate_feedback.py (1)
GenerateSuggestionsPrompt(78-86)athena/modules/text/module_text_llm/module_text_llm/llm_as_profiler/prompt_profiler.py (1)
ProfilerPrompt(51-60)athena/modules/text/module_text_llm/module_text_llm/llm_as_profiler/generate_suggestions.py (1)
generate_suggestions(18-128)athena/athena/athena/schemas/learner_profile.py (1)
LearnerProfile(5-97)
⏰ Context from checks skipped due to timeout of 90000ms (13)
- GitHub Check: build_image (athena/assessment_module_manager)
- GitHub Check: build_image (athena/modules/modeling/module_modeling_llm)
- GitHub Check: build_image (athena/modules/programming/module_programming_winnowing)
- GitHub Check: build_image (athena/modules/text/module_text_cofee)
- GitHub Check: build_image (athena/modules/text/module_text_llm)
- GitHub Check: build_image (athena/modules/programming/module_programming_apted)
- GitHub Check: build_image (athena/playground)
- GitHub Check: build_image (athena/modules/programming/module_example)
- GitHub Check: build_image (athena/log_viewer)
- GitHub Check: build_image (athena/modules/programming/module_programming_llm)
- GitHub Check: build_image (athena/modules/programming/module_programming_themisml)
- GitHub Check: test
- GitHub Check: prospector
🔇 Additional comments (10)
athena/modules/text/module_text_llm/module_text_llm/config.py (1)
10-10: LGTM!The integration of
LLMAsProfilerConfiginto the configuration framework follows the established pattern and is implemented correctly.Also applies to: 13-13
athena/modules/text/module_text_llm/module_text_llm/llm_as_profiler/prompt_generate_feedback.py (3)
1-4: LGTM! Clean imports.The imports are well-organized and include all necessary components for the Pydantic models and enums.
90-93: LGTM! Well-defined enum for suggested actions.The
SuggestedActionenum is clearly defined with descriptive values that align with the pedagogical approach described in the system message.
121-124: LGTM! Clean aggregation model.The
AssessmentModelprovides a clear structure for collecting multiple feedback items.athena/modules/text/module_text_llm/module_text_llm/llm_as_profiler/prompt_profiler.py (6)
1-4: LGTM! Proper imports.The imports are correctly ordered and include all necessary components.
6-39: Excellent educational framework alignment.The system message demonstrates strong pedagogical understanding by:
- Properly implementing Bloom's taxonomy cognitive levels
- Providing clear competency evaluation criteria
- Emphasizing evidence-based assessment
The instructions are comprehensive and educationally sound.
65-69: LGTM! Well-ordered competency status enum.The
CompetencyStatusenum is logically ordered from lowest to highest demonstration level, which makes sense for evaluation progression.
72-78: Excellent Bloom's taxonomy implementation.The
CognitiveLevelenum correctly implements Bloom's taxonomy in the proper hierarchical order from basic recall to complex creation.
81-92: LGTM! Comprehensive competency definition.The
RequiredCompetencymodel effectively captures the essential elements needed for competency tracking with proper optional fields.
112-116: LGTM! Clean profile aggregation.The
SubmissionCompetencyProfileprovides a clear structure for collecting all competency evaluations.
athena/modules/text/module_text_llm/module_text_llm/llm_as_profiler/__init__.py
Show resolved
Hide resolved
athena/modules/text/module_text_llm/module_text_llm/llm_as_profiler/generate_suggestions.py
Outdated
Show resolved
Hide resolved
athena/modules/text/module_text_llm/module_text_llm/llm_as_profiler/generate_suggestions.py
Show resolved
Hide resolved
athena/modules/text/module_text_llm/module_text_llm/llm_as_profiler/generate_suggestions.py
Outdated
Show resolved
Hide resolved
athena/modules/text/module_text_llm/module_text_llm/llm_as_profiler/generate_suggestions.py
Outdated
Show resolved
Hide resolved
athena/modules/text/module_text_llm/module_text_llm/llm_as_profiler/generate_suggestions.py
Show resolved
Hide resolved
athena/modules/text/module_text_llm/module_text_llm/llm_as_profiler/prompt_generate_feedback.py
Outdated
Show resolved
Hide resolved
LeonWehrhahn
temporarily deployed
to
Athena - Test 1
GitHub Actions
Inactive
github-actions
bot
added
lock:athena-test1
LeonWehrhahn
removed
the
lock:athena-test1
|
There hasn't been any activity on this pull request recently. Therefore, this pull request has been automatically marked as stale and will be closed if no further activity occurs within seven days. Thank you for your contributions. |
|
There hasn't been any activity on this pull request recently. Therefore, this pull request has been automatically marked as stale and will be closed if no further activity occurs within seven days. Thank you for your contributions. |
Motivation and Context
As we want to generate more personalized feedback, we want to introduce an approach that would make use of the LLM to profile the students using their submissions.
Description
Adds a new feedback generation approach, which in the first iteration profiles the student. Then uses that profile to generate personalized feedback.
1. Competency-Based Profiling
2. Personalized Feedback Generation
Next steps
Steps for Testing
Playground
Artemis
athena/modules/text/module_text_llm/module_text_llm/config.py:Testserver States
Note
These badges show the state of the test servers.
Green = Currently available, Red = Currently locked
Click on the badges to get to the test servers.
Screenshots
Competency Schemas
New Feedback Attribute: Suggested Action
Summary by CodeRabbit
Summary by CodeRabbit