Points: 2-3 days
Description: Support return hidden states in serving_chat.py and serving_completions.py
Deliverables:
Task: Missing hidden states **
Current streaming logic in adapter.py#L691-699:
|
hidden_states = None |
|
if isinstance(request, list) and request[idx].return_hidden_states: |
|
hidden_states = ret_item["meta_info"].get("hidden_states", None) |
|
elif (not isinstance(request, list)) and request.return_hidden_states: |
|
hidden_states = ret_item["meta_info"].get("hidden_states", None) |
|
if hidden_states is not None: |
|
hidden_states = ( |
|
hidden_states[-1] if hidden_states and len(hidden_states) > 1 else [] |
|
) |
Current logic in adapter.py#L1370-L1382:
|
if isinstance(request, list) and request[idx].return_hidden_states: |
|
include_hidden_states = True |
|
elif not isinstance(request, list) and request.return_hidden_states: |
|
include_hidden_states = True |
|
else: |
|
include_hidden_states = False |
|
if include_hidden_states and ret_item["meta_info"].get("hidden_states", None): |
|
hidden_states = ret_item["meta_info"]["hidden_states"] |
|
hidden_states = ( |
|
hidden_states[-1] if hidden_states and len(hidden_states) > 1 else [] |
|
) |
|
else: |
|
hidden_states = None |
- Need to support it in
serving_chat.py and serving_completions.py .
Points: 2-3 days
Description: Support return hidden states in
serving_chat.pyandserving_completions.pyDeliverables:
Task: Missing hidden states **
Current streaming logic in
adapter.py#L691-699:sglang/python/sglang/srt/openai_api/adapter.py
Lines 691 to 699 in ca92911
Current logic in
adapter.py#L1370-L1382:sglang/python/sglang/srt/openai_api/adapter.py
Lines 1370 to 1382 in ca92911
serving_chat.pyandserving_completions.py.