HPE puts storage to work on AI’s expensive memory problem

HPE is giving its Alletra Storage MP X10000 another job: holding inference state that would otherwise be evicted from expensive GPU memory and recalculated later. In testing validated by Kamiwaza using NVIDIA H200 GPUs, the Kamiwaza AI platform, and HPE Alletra MP X10000, the solution achieved up to 20× faster time to first token and 17× higher effective inference throughput versus running without KV cache offload. The GPUs did not become cleverer.