[Bug] HuggingFace and SGLang inference don't match

[pratcooper]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 5. Please use English, otherwise it will be closed.

Describe the bug

The accuracy of the model is degraded due to inconsistent outputs from SGLang. While HF and vLLM produce consistent results such as "A" or "B," SGLang occasionally outputs responses like "I can't process that request." or "A." / "B." This inconsistency impacts overall accuracy.

Reproduction

What command or script did you run?
A script for generating outputs using a LLaMA 3.1 8B Istruct model with LoRA.

Which model are you using?
LLaMA 3.1 with LoRA applied.

Steps to reproduce:

1. Run the script with HF, vLLM, and SGLang configurations.
2. Compare the single-token outputs between the frameworks.
3. Observe the inconsistent behavior in SGLang.

---

Hugging Face Code Snippet :

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = LlamaForCausalLM.from_pretrained(
model_name,
load_in_8bit=False,
torch_dtype=torch.float16,
device_map='auto',
)
adaptor_path = './model_spec/checkpoints/checkpoint-200-vllm'
model = PeftModel.from_pretrained(
model,
adaptor_path,
torch_dtype=torch.float16,
)

model.config.pad_token_id = tokenizer.pad_token_id = 0
model.config.bos_token_id = 1
model.config.eos_token_id = 2
model.generation_config.pad_token_id = tokenizer.pad_token_id
model.eval()

def evaluate(
instruction,
input=None,
temperature=0,
top_p=1,
top_k=-1,
num_beams=4,
max_new_tokens=128,
stream_output=False,
**kwargs,
):
prompt = f"<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n{instruction}<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n{input}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"

inputs = tokenizer(prompt, return_tensors="pt")
input_ids = inputs["input_ids"].to('cuda')
generation_config = GenerationConfig(
    temperature=temperature,
    top_p=top_p,
    top_k=top_k,
    num_beams=num_beams,
    **kwargs,
)

with torch.no_grad():
    generation_output = model.generate(
        input_ids=input_ids,
        generation_config=generation_config,
        return_dict_in_generate=True,
        output_scores=True,
        max_new_tokens=max_new_tokens
    )
s = generation_output.sequences[0]
output = tokenizer.decode(s, skip_special_tokens=True)
result = output.split('assistant')[1].strip()
return result

---

SGLang Code Snippet :

import sglang as sgl
from sglang import *
import logging
import json
import torch

from typing import Union, List
from vllm import LLM, SamplingParams
from vllm.inputs import TokensPrompt
from vllm.lora.request import LoRARequest
from collections import defaultdict

from utils.usecase_prompts import UseCasePrompter
from utils.lora_adapters import UseCaseLoraAdapters
from utils.prompter import Prompter

logging.basicConfig(format='%(asctime)s %(message)s')
logger = logging.getLogger()
logger.setLevel(logging.INFO)

class SimpleSGLangLlama2:
def init(self, base_model_path, number_of_gpu=1, gpu_memory_utilization=0.4):
self.base_model_path = base_model_path
self._model = sgl.Engine(model_path=self.base_model_path)

def generate(
    self,
    prompt: Union[str, List[int]],
    temperature: float = 0.0,
    top_p: float = 1.0,
    top_k: int = -1,
    use_beam_search: bool = True,
    max_new_tokens: int = 128,
    best_of: int = 4
) -> List[str]:
    sampling_params = {"temperature": temperature, "top_p": top_p, "top_k":top_k, "max_new_tokens": max_new_tokens}
    # Build final_prompt_text, then:
    outputs = self._model.generate(
        [final_prompt_text],
        sampling_params,
        lora_path=adaptor_path
    )
    results = [output['text'] for output in outputs]
    return results

if name == "main":
model_path = "./models/meta-llama/Meta-Llama-3.1-8B-Instruct"
llm = SimpleSGLangLlama2(model_path)

---

Environment

Run the class in notebooks environment.

SGLang 0.4.0 (with flashinfer 0.1.6+cu121torch2.4)

[pratcooper]

@Ying1123

[Fridge003]

@pratcooper Hi, we just updated the codes for Lora. The misalignment bug should be fixed. Would you please test it?

[zhyncs]

BTW I'll release a new version soon

[Fridge003]

@pratcooper Hi, we just updated the codes for Lora. The misalignment bug should be fixed. Would you please test it?

Also, the lora paths of adaptors should be passed in when launching server (engine). So the line of launching engine in script should be modified to

self._model = sgl.Engine(model_path=self.base_model_path, lora_paths=[adaptor_path])

to make it work.

[pratcooper]

@Fridge003 Thanks ! I will try to test it this week. Can you please provide me version number for this fix ?

[Fridge003]

sglang 0.4.2.post4, sgl-kernel 0.0.3.post3, flashinfer 0.2.0.post2

[Fridge003]

Hi @pratcooper Is this issue fixed?

[pratcooper]

@Fridge003 Have been a little occupied lately. Will try to get back as soon as possible. Sorry for delay.

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.