fix(core): implement proper token counting for mixed content in getNumTokens

[christian-bromann]

Description

This PR fixes a critical issue in the BaseLanguageModel.getNumTokens() method where it would return 0 tokens for any non-string MessageContent, effectively ignoring text content in mixed-content messages.

Problem

Previously, when calling getNumTokensFromMessages() with messages containing mixed content (e.g., text + image_url arrays), the text portions were not being counted, leading to inaccurate token usage calculations. The method had a TODO comment and simply returned 0 for any array content:

// TODO: Figure out correct value.
if (typeof content !== "string") {
  return 0;
}

Solution

Implemented proper text extraction logic that:

• Maintains backward compatibility: String content works exactly as before
• Extracts text from arrays: Processes MessageContentComplex[] to find and concatenate all text blocks
• Ignores non-text content: Image URLs, audio, and other media content are properly ignored for token counting
• Handles edge cases: Empty arrays, image-only content, and multiple text blocks

Example

const messages = [
  new HumanMessage({
    content: [
      {
        type: "text",
        text: "What is this image?", // This text is now properly counted
      },
      {
        type: "image_url",
        image_url: {
          url: "https://example.com/image.jpg",
        },
      },
    ],
  }),
];

// Before: getNumTokensFromMessages would undercount due to 0 tokens from getNumTokens
// After: Correctly counts tokens for "What is this image?"

Changes Made

• Core fix: Replaced TODO implementation in getNumTokens() with proper text extraction
• Comprehensive tests: Added test suite covering mixed content, edge cases, and compatibility
• Test restructuring: Improved test organization with describe/it blocks for better clarity

Testing

• ✅ All existing tests pass (backward compatibility maintained)
• ✅ New tests cover mixed content scenarios
• ✅ Edge cases tested (empty arrays, image-only content, multiple text blocks)
• ✅ Integration with getNumTokensFromMessages verified

This fix ensures accurate token counting for modern chat applications that use rich message content with text, images, and other media types.

Fixes #8310
Fixes #8336

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
langchainjs-docs	✅ Ready (Inspect)	Visit Preview		Jun 17, 2025 1:02am

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchainjs-api-refs	⬜️ Ignored (Inspect)			Jun 17, 2025 1:02am

[hntrl]

Better than what we had before but I don't know if this is "end-state" token counting (for instance in python we consider other things like function calls apart of the token count, there's probably other ones we should be considering). Probably at a minimum we re-add the comment to signal a more appropriate solution is needed. It would be interesting to compare different message content instances with gpt-2 tokenizer and this fallback method.

I'll also call out I don't think the blast radius on this is big (I think) since we only use this in the trimMessages utility

https://github.com/langchain-ai/langchain/blob/master/libs/core/langchain_core/messages/utils.py#L86

[christian-bromann]

Thanks for reviewing!

I'll also call out I don't think the blast radius on this is big (I think) since we only use this in the trimMessages utility

There have been 2 reported instances of this failing for users, see #8310 and #8336. I agree with you, this is definitely not perfect but addresses the issues reported. I've added a comment to revisit this section at a later time. Let me know if there is anything else I can do.

[hntrl]

thanks @christian-bromann!