KV Cache Context Memory Estimator
Estimate inference KV cache from layers, KV heads, head dimension, context, batch, and data type.
Parameters
Results
Assumptions
- 采用標準自回歸注意力 K/V 雙緩存結構。
Important notes
- Pagination緩存、量化、滑動窗口、張量并行和實現開銷會改變實際顯存。
Estimate inference KV cache from layers, KV heads, head dimension, context, batch, and data type.