Added tokenize keyword arguments to feature extraction pipeline

[quancore]

What does this PR do?

The PR adds keyword arguments for the tokenizer for the feature extraction pipeline. Fixes: #19374

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@LysandreJik
@Narsil

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[Narsil]

Thanks for this.

Thanks for the tests too !
I added a remark to keep from breaking anything in user code.

[Narsil]

Unfortunately we have to keep truncation in there since it was already allowed in order to NOT break anything.

Something like:

def _sanitize_parameters(self, truncation=None, tokenize_kwargs=None, **kwargs):
    # handle tokenize_kwargs first
    if truncation is not None:
         if 'truncation' in tokenize_kwargs:
             raise ValueError("This is defined twice")
         tokenize_kwargs["truncation"] = truncation
    

[quancore]

@Narsil But the problem is truncation is already a parameter of the tokenizer, why the parameter should be kept separately?

[Narsil]

Because it was there before, hence users might (and probably HAVE) started using it.
And since we cannot break user code, we have to keep backward compatibility with it.

[Narsil]

After checking, the broken tests are exactly broken by the lack of truncation support.

Also for quality you should be able to to

pip install -e .[dev] #  or pip install transformers[dev]
make fixup

Cheers.

[quancore]

@Narsil I made the changes you indicate.

[Narsil]

LGTM.

@sgugger

[sgugger]

A couple of nits, but LGTM otherwise! Thanks a lot for working on this!

[quancore]

@sgugger I have moved the import to top.

[sgugger]

Thanks a lot!