You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
DefaultPromptHandler downloads unnecessary GPT2 tokenizer to a read-only filesystem (eg. AWS Lambda), when truncate argument is set to False.
# Truncate prompt if prompt tokens > model_max_length-max_length
# (max_length is the length of the generated text)
# we use GPT2 tokenizer which will likely provide good token count approximation
self.prompt_handler=DefaultPromptHandler(
tokenizer="gpt2",
model_max_length=model_max_length,
max_length=self.max_lengthor100,
)
Describe the solution you'd like
If truncate is set to false, GPT2 tokenizer for token estimation shouldn't be loaded.
Describe alternatives you've considered
Simply place a condition to only load the tokenizer if truncate is set to true.
# Truncate prompt if prompt tokens > model_max_length-max_length# (max_length is the length of the generated text)# we use GPT2 tokenizer which will likely provide good token count approximation--->ifself.truncate:
self.prompt_handler=DefaultPromptHandler(
tokenizer="gpt2",
model_max_length=model_max_length,
max_length=self.max_lengthor100,
)
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
DefaultPromptHandler downloads unnecessary GPT2 tokenizer to a read-only filesystem (eg. AWS Lambda), when
truncate
argument is set toFalse
.Check
haystack-core-integrations/integrations/amazon_bedrock/src/haystack_integrations/components/generators/amazon_bedrock/generator.py
Lines 148 to 156 in cf52ce9
Describe the solution you'd like
If truncate is set to false, GPT2 tokenizer for token estimation shouldn't be loaded.
Describe alternatives you've considered
Simply place a condition to only load the tokenizer if truncate is set to true.
The text was updated successfully, but these errors were encountered: