Token count mismatch with Google AI Studio

[evrenesat]

Thanks for creating and maintaining repomix.

With o200k_base, repomix reports "969,990 tokens", but AI Studio claims it's "1,156,615". Is this a known issue with AI Studio? If not, is there a way to configure repomix to be compatible with it?

Btw, cl100k_base and o200k_base returns same token counts, is this expected?

[yamadashy]

Hi, @evrenesat !
Thanks for raising this — yes, this is a known limitation.

Currently, Repomix only supports OpenAI-compatible tokenizers, so token counts may not match with Claude or Gemini models.

For Claude, you can get an accurate token count using the Claude SDK. As for Gemini, it seems there isn't a publicly available tokenizer for JavaScript yet, so that's something we're still figuring out.