… We therefore call for inference papers and benchmarks to report Joules/token, active binding constraint, PUE-adjusted delivered power, and utilization-adjusted token output alongside accuracy and latency. …
… Across in-domain and out-of-domain evaluation, multi-token DiffRetriever substantially improves over single-token on every diffusion backbone we test, while autoregressive multi-token is flat or negative and pays a latency cost that scales with K where diffusion does not. …
Papers arxiv:2605.06105 Shallow Prefill, Deep Decoding: Efficient Long-Context Inference via Layer-Asymmetric KV Visibility Published on May 7 Submitted by jeongseokoh on May 11 Authors: Jungsuk Oh , , , , Abstract SPEED is a phase-asymmetric KV-visibility policy that reduces long-context inference… …
… AI-generated summary DeepSeek Sparse Attention DSA sets the state of the art for fine-grained inference-time sparse attention by introducing a learned token-wise indexer that scores every prefix token and selects the most relevant ones for the main attention. …
Papers arxiv:2605.09649 Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction Published on May 10 Submitted by Ngoc Bui on May 12 Yale University Authors: , , , Abstract Learned global retention-based key-value cache eviction improves long-context reasoning by sel… …
… The prevailing practice typically adopts global encoding followed by post-ViT compression . Global encoding produces massive token sequences, while post-ViT compression incurs the full quadratic attention cost of the ViT before any token reduction takes place. …
Papers arxiv:2605.12825 Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion Published on May 12 Submitted by Nguyen Van Chien on May 14 Authors: Chien Van Nguyen , , , , Franck Dernoncourt , Abstract Orthrus is a dual-architecture framework that combines autoregressive LLMs … …
… AI-generated summary On-policy distillation offers dense, per-token supervision for training reasoning models; however, it remains unclear under which conditions this signal is beneficial and under which it is detrimental. …
… This preserves tool-call fidelity, gives cross-round consistency, and locks in first-turn protection even if the on-device LLM is later compromised. 📊 Results Qwen3-4B + Gemini 3 Flash 📈 +15-36% accuracy and 2-6× lower leakage vs SOTA device-cloud baselines on $\tau^2$-Bench Airline/Retail and GAIA… …
… I also recently read a related guide: https://mobisoftinfotech.com/resources/blog/ai‑development/llm‑api‑pricing‑guide — which gives practical advice on LLM API usage, token‑based pricing, and how to plan costs when working with LLMs. …