Additional Trainer Argument for features of different modalities #3225
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…he previous float32
…feauture keys (e.g for other modalities) to the collect features function.
|
Hello! I think this is indeed quite valuable to support, so I'd like to help you out here. However, I think we might be better off going in a different direction that should work out of the box without the user having to ever specify anything: #3276 Feel free to give me some feedback there! Also, apologies for the delays. I've been focusing on the huge #3222, which is now nearing completion.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi,
I have been working on multi-modal embedding models based on the sentence-transformers framework and found that I need to hard code keys into the "collect_features" method within the trainer. I simply added an argument to allow passing a list of additional keys that might be present in your features (similar to "pixel_values" for CLIP). I would appreciate a merge so I don't need to rely on my fork for the multi-modal embedding model.
Best,
Jonatan