✨ feat: add image generation support for google provider #8503
+434
−15
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
Reviewer's GuideThis PR adds end-to-end support for image generation via Google’s Imagen API by implementing a new createImage method in LobeGoogleAI, enriching model configurations with Imagen model entries, extending runtime types and error utilities for image generation errors, and validating the new feature through extensive tests. Sequence diagram for createImage flow in LobeGoogleAIsequenceDiagram
participant User
participant LobeGoogleAI
participant GoogleAIClient
User->>LobeGoogleAI: createImage(payload)
LobeGoogleAI->>GoogleAIClient: generateImages({model, prompt, config})
GoogleAIClient-->>LobeGoogleAI: {generatedImages: [{image: {imageBytes}}]}
LobeGoogleAI-->>User: {imageUrl}
Note over LobeGoogleAI: On error, throws AgentRuntimeError.createImage
Class diagram for AgentRuntimeError utility with image error supportclassDiagram
class AgentRuntimeError {
+chat(error: ChatCompletionErrorPayload): ChatCompletionErrorPayload
+init(errorType: ILobeAgentRuntimeErrorType | string | number, error?: any): AgentInitErrorPayload
+createImage(error: CreateImageErrorPayload): CreateImageErrorPayload
+textToImage(error: any): any
}
class CreateImageErrorPayload {
+endpoint?: string
+error: object
+errorType: ILobeAgentRuntimeErrorType
+provider: string
}
AgentRuntimeError --> CreateImageErrorPayload
File-Level Changes
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
👍 @tjx666 Thank you for raising your pull request and contributing to our Community |
TestGru AssignmentSummary
Tip You can |
dosubot
bot
added
size:L
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey @tjx666 - I've reviewed your changes - here's some feedback:
- Instead of hard-coding numberOfImages to 1 in createImage, consider making it a configurable parameter so users can request multiple images if needed.
- Rather than using console.error for error logging in createImage, switch to your project’s existing logging utility to keep logs consistent and more easily configurable.
- createImage assumes a PNG output when building the data URI—consider inferring or allowing configuration of the MIME type in case other formats are supported in the future.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- Instead of hard-coding numberOfImages to 1 in createImage, consider making it a configurable parameter so users can request multiple images if needed.
- Rather than using console.error for error logging in createImage, switch to your project’s existing logging utility to keep logs consistent and more easily configurable.
- createImage assumes a PNG output when building the data URI—consider inferring or allowing configuration of the MIME type in case other formats are supported in the future.
## Individual Comments
### Comment 1
<location> `src/libs/model-runtime/google/index.ts:274` </location>
<code_context>
+ }
+
+ const imageBytes = generatedImage.image.imageBytes;
+ const imageUrl = `data:image/png;base64,${imageBytes}`;
+
+ return { imageUrl };
</code_context>
<issue_to_address>
The image MIME type is hardcoded as PNG, which may not always match the actual image format.
If other image formats are possible, make the MIME type dynamic or validate the format. Otherwise, document the PNG-only assumption.
</issue_to_address>
<suggested_fix>
<<<<<<< SEARCH
const imageBytes = generatedImage.image.imageBytes;
const imageUrl = `data:image/png;base64,${imageBytes}`;
return { imageUrl };
=======
const imageBytes = generatedImage.image.imageBytes;
// Determine MIME type dynamically if available, otherwise assume PNG
let mimeType = 'image/png';
if (generatedImage.image.mimeType) {
mimeType = generatedImage.image.mimeType;
} else {
// If only PNG is supported, document this assumption
// Assumes generated images are always PNG. Update this if other formats are supported.
}
const imageUrl = `data:${mimeType};base64,${imageBytes}`;
return { imageUrl };
>>>>>>> REPLACE
</suggested_fix>
### Comment 2
<location> `src/config/aiModels/google.ts:501` </location>
<code_context>
-export const allModels = [...googleChatModels];
+// Common parameters for Imagen models
+const imagenBaseParameters: ModelParamsSchema = {
+ aspectRatio: {
+ default: '1:1',
+ enum: ['1:1', '16:9', '9:16', '3:4', '4:3'],
+ },
+ prompt: { default: '' },
+};
+
</code_context>
<issue_to_address>
The 'prompt' parameter in imagenBaseParameters lacks validation or description.
Add a description and validation, such as minLength, to the 'prompt' parameter for better schema clarity and validation.
</issue_to_address>
<suggested_fix>
<<<<<<< SEARCH
prompt: { default: '' },
=======
prompt: {
default: '',
description: 'The text prompt describing the image to generate.',
minLength: 1,
},
>>>>>>> REPLACE
</suggested_fix>
### Comment 3
<location> `src/libs/model-runtime/types/type.ts:19` </location>
<code_context>
provider: string;
}
+export interface CreateImageErrorPayload {
+ [key: string]: any;
+ endpoint?: string;
+ error: object;
+ errorType: ILobeAgentRuntimeErrorType;
+ provider: string;
+}
+
</code_context>
<issue_to_address>
The CreateImageErrorPayload interface allows arbitrary keys, which may reduce type safety.
Consider limiting the interface to specific fields or clearly documenting its structure to improve maintainability and type safety.
</issue_to_address>
<suggested_fix>
<<<<<<< SEARCH
export interface CreateImageErrorPayload {
[key: string]: any;
endpoint?: string;
error: object;
errorType: ILobeAgentRuntimeErrorType;
provider: string;
}
=======
export interface CreateImageErrorPayload {
endpoint?: string;
error: object;
errorType: ILobeAgentRuntimeErrorType;
provider: string;
}
>>>>>>> REPLACE
</suggested_fix>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Comment on lines
273
to
279
| const imageBytes = generatedImage.image.imageBytes; | ||
| const imageUrl = `data:image/png;base64,${imageBytes}`; | ||
|
|
||
| return { imageUrl }; |
There was a problem hiding this comment.
suggestion: The image MIME type is hardcoded as PNG, which may not always match the actual image format.
If other image formats are possible, make the MIME type dynamic or validate the format. Otherwise, document the PNG-only assumption.
Suggested change
| const imageBytes = generatedImage.image.imageBytes; | |
| const imageUrl = `data:image/png;base64,${imageBytes}`; | |
| return { imageUrl }; | |
| const imageBytes = generatedImage.image.imageBytes; | |
| // Determine MIME type dynamically if available, otherwise assume PNG | |
| let mimeType = 'image/png'; | |
| if (generatedImage.image.mimeType) { | |
| mimeType = generatedImage.image.mimeType; | |
| } else { | |
| // If only PNG is supported, document this assumption | |
| // Assumes generated images are always PNG. Update this if other formats are supported. | |
| } | |
| const imageUrl = `data:${mimeType};base64,${imageBytes}`; | |
| return { imageUrl }; |
| default: '1:1', | ||
| enum: ['1:1', '16:9', '9:16', '3:4', '4:3'], | ||
| }, | ||
| prompt: { default: '' }, |
There was a problem hiding this comment.
suggestion: The 'prompt' parameter in imagenBaseParameters lacks validation or description.
Add a description and validation, such as minLength, to the 'prompt' parameter for better schema clarity and validation.
Suggested change
| prompt: { default: '' }, | |
| prompt: { | |
| default: '', | |
| description: 'The text prompt describing the image to generate.', | |
| minLength: 1, | |
| }, |
src/libs/model-runtime/types/type.ts
Outdated
Comment on lines
19
to
23
| export interface CreateImageErrorPayload { | ||
| [key: string]: any; | ||
| endpoint?: string; | ||
| error: object; | ||
| errorType: ILobeAgentRuntimeErrorType; | ||
| provider: string; | ||
| } |
There was a problem hiding this comment.
suggestion: The CreateImageErrorPayload interface allows arbitrary keys, which may reduce type safety.
Consider limiting the interface to specific fields or clearly documenting its structure to improve maintainability and type safety.
Suggested change
| export interface CreateImageErrorPayload { | |
| [key: string]: any; | |
| endpoint?: string; | |
| error: object; | |
| errorType: ILobeAgentRuntimeErrorType; | |
| provider: string; | |
| } | |
| export interface CreateImageErrorPayload { | |
| endpoint?: string; | |
| error: object; | |
| errorType: ILobeAgentRuntimeErrorType; | |
| provider: string; | |
| } |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #8503 +/- ##
==========================================
+ Coverage 85.31% 85.33% +0.01%
==========================================
Files 908 908
Lines 68547 68609 +62
Branches 4434 4672 +238
==========================================
+ Hits 58483 58545 +62
Misses 10064 10064
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
- Introduced `createImage` method in `LobeGoogleAI` for generating images based on user prompts and aspect ratios. - Added error handling for various scenarios in image generation. - Updated model configurations to include new image models in `google.ts`. - Enhanced test coverage for image generation functionality in `index.test.ts`.
tjx666
force-pushed
the
tj/feat/google-ai-image
branch
from
fd86a38 to
723bb79
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
createImagemethod inLobeGoogleAIfor generating images based on user prompts and aspect ratios.google.ts.index.test.ts.💻 变更类型 | Change Type
🔀 变更说明 | Description of Change
📝 补充信息 | Additional Information
Summary by Sourcery
Add image generation support for the Google provider by implementing a createImage method in LobeGoogleAI, updating model configurations with new Imagen models, and extending tests to cover image generation scenarios.
New Features:
Enhancements:
Tests: