Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: set DEFAULT_CONTEXT_LENGTH to 8192 #4963

Merged
merged 1 commit into from
Apr 3, 2025

Conversation

pppy2012
Copy link
Contributor

@pppy2012 pppy2012 commented Apr 3, 2025

Change the DEFAULT_CONTEXT_LENGTH from 8096 to 8192, the reason is to enhance compatibility between continue and ollama to improve performance.
Key reasons:

  • Ollama uses 8192 as the default context length.
  • Ollama will reload the model if the context changes even the model is same.
  • The ollama backend may reload models to support continue's requests with 8096 context length.
  • Likewise, models preloaded for ​continue may be evicted by other requests using the default context length.
  • Frequent model reloading introduces significant performance overhead.

So, it's better to use same default context length to improve the performance in default environment.

@pppy2012 pppy2012 requested a review from a team as a code owner April 3, 2025 02:43
@pppy2012 pppy2012 requested review from tomasz-stefaniak and removed request for a team April 3, 2025 02:43
Copy link

netlify bot commented Apr 3, 2025

Deploy Preview for continuedev canceled.

Name Link
🔨 Latest commit f163768
🔍 Latest deploy log https://app.netlify.com/sites/continuedev/deploys/67edf5d6c1c10d0008f4b094

@sestinj sestinj merged commit add7a13 into continuedev:main Apr 3, 2025
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants