{"slug":"fast-inference","label":"Fast Inference","item_count":3,"day_count":3,"source_count":2,"first_seen":"2026-10-26T17:06:14+00:00","last_updated":"2026-11-10T22:28:15+00:00","generated_at":"2026-11-11T00:14:41.869679+00:00","sources":["arxiv_cs_cl","hackernews_ai"],"days":[{"date":"2026-10-26","items":[{"title":"Show HN: Resin, fast native scheduler for LLM inference engines","url":"https://github.com/ekzhang/resin","source":"hackernews_ai","type":"news","summary_1line":"Show HN: Resin, fast native scheduler for LLM inference engines","sid":"0a5f683f2cf3f23c","published":"2026-10-26T17:06:14+00:00","editor_note":"Opens the thread with a scheduling-layer approach to faster serving."}]},{"date":"2026-11-07","items":[{"title":"Hibana: Fast Agentic Inference on Unified-Memory Devices","url":"http://arxiv.org/abs/2611.10469v1","source":"arxiv_cs_cl","type":"paper","summary_1line":"Local AI promises capable personal assistants that run on users' own devices, keeping data and computation under their control. Open-weight models that fit on local devices can now handle practical tasks such as codin...","why_it_matters":"Matches feed focus: agentic.","sid":"a38548abbeed9a82","published":"2026-11-07T04:38:35+00:00","editor_note":"Moves the thread from servers to local devices, with a research take on agent workloads."}]},{"date":"2026-11-10","items":[{"title":"Show HN: Warmctl – Fast cold starts for vLLM inference","url":"https://github.com/TensorRails/warmctl","source":"hackernews_ai","type":"news","summary_1line":"Show HN: Warmctl – Fast cold starts for vLLM inference","sid":"131509a7651f4930","published":"2026-11-10T22:28:15+00:00","editor_note":"Adds cold-start latency as a separate bottleneck, specific to vLLM deployments."}]}],"editorial":{"tldr":"This is a loose cluster of three unrelated projects that share one goal: serving LLMs faster below the model layer. It began with Resin, a native scheduler for inference engines, and was followed by Hibana, an arXiv paper on agentic inference for unified-memory devices.","stale":false,"whats_new":"Warmctl, a new open-source tool shared on Show HN, targets slow cold starts for vLLM servers, adding a startup-latency fix to a thread about scheduling and local serving.","why_it_matters":"Inference latency increasingly depends on the layers around the model (request scheduling, memory layout, server warm-up), and each of these projects targets one of those layers.","take_for_builders":"If vLLM startup time limits your autoscaling, measure your current cold-start time first, then trial Warmctl on a non-production replica and compare.","beats":[{"kicker":"SCHEDULING","tone":"launch","headline":"Resin: a native scheduler for LLM inference engines","summary":"A Show HN project that puts request scheduling into a native component sitting in front of the inference engine.","sids":["0a5f683f2cf3f23c"]},{"kicker":"LOCAL","tone":"rising","headline":"Hibana targets agentic inference on unified-memory devices","summary":"Research on running agent workloads such as coding with open-weight models on local hardware with shared CPU/GPU memory.","sids":["a38548abbeed9a82"]},{"kicker":"NOW","tone":"now","headline":"Warmctl tackles vLLM cold starts","summary":"A Show HN tool aimed at the startup delay before a vLLM server can take traffic.","sids":["131509a7651f4930"]}],"open_questions":["Does Warmctl publish cold-start timings against a stock vLLM startup on named models and GPUs?","Does Resin integrate with vLLM or SGLang, or does it require its own engine?","Does Hibana release code, and which unified-memory hardware does it benchmark on?"],"generated_at":"2026-11-11T00:13:33Z"}}