Try this oneMedium8 minGitHub
The KV Cache OOM
An A100 server has 51 active chats and 13 GiB free, yet a new request is killed. Switch the allocator and send the request.
- In use by a chat
- Free
- New request
- node
- a100-inference-03
- active chats
- 51
- gpu mem free
- 13.25 GiB (13 of 48 blocks)
- status
- idle, waiting for a request
A passing run records