Skip to content

AmazonBedrockGenerator shouldn't load tokenizer if truncate is set to False. #1125

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
hypathia opened this issue Oct 4, 2024 · 0 comments · Fixed by #1152
Closed

AmazonBedrockGenerator shouldn't load tokenizer if truncate is set to False. #1125

hypathia opened this issue Oct 4, 2024 · 0 comments · Fixed by #1152
Assignees
Labels

Comments

@hypathia
Copy link

hypathia commented Oct 4, 2024

Is your feature request related to a problem? Please describe.
DefaultPromptHandler downloads unnecessary GPT2 tokenizer to a read-only filesystem (eg. AWS Lambda), when truncate argument is set to False.

Check

# Truncate prompt if prompt tokens > model_max_length-max_length
# (max_length is the length of the generated text)
# we use GPT2 tokenizer which will likely provide good token count approximation
self.prompt_handler = DefaultPromptHandler(
tokenizer="gpt2",
model_max_length=model_max_length,
max_length=self.max_length or 100,
)

Describe the solution you'd like
If truncate is set to false, GPT2 tokenizer for token estimation shouldn't be loaded.

Describe alternatives you've considered
Simply place a condition to only load the tokenizer if truncate is set to true.

        # Truncate prompt if prompt tokens > model_max_length-max_length
        # (max_length is the length of the generated text)
        # we use GPT2 tokenizer which will likely provide good token count approximation
--->    if self.truncate:
            self.prompt_handler = DefaultPromptHandler(
                tokenizer="gpt2",
                model_max_length=model_max_length,
                max_length=self.max_length or 100,
            )
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants