# llm-reasoning Published articles for llm-reasoning. This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Claude Fable 5.1: Benchmark results and reasoning-effort experiments DevFeed: [Claude Fable 5.1: Benchmark results and reasoning-effort experiments]() Original publisher: [Read original article]() Author: Simon Willison Published: 2026-09-01T23:57:28Z Content type: article Language: en Sources: [Simon Willison]() Topics: [Fable](), [Benchmark](), [coding](), [Large Language Model](), [SVG]() Tags: [ai](), [ai-2-235](), [anthropic](), [anthropic-336](), [benchmark](), [benchmarks](), [claude](), [claude-310](), [coding](), [fable](), [generative-ai](), [generative-ai-1-981](), [llm](), [llm-reasoning](), [llm-reasoning-103](), [llm-release](), [llm-release-231](), [llms](), [llms-1-947](), [models](), [pelican-riding-a-bicycle](), [pelican-riding-a-bicycle-142](), [reasoning](), [svg]() ### AI overview The article examines Claude Fable 5.1 through benchmark results and a pelican SVG-generation experiment across its five reasoning-effort levels. It reports that low and medium settings appeared to skip reasoning for this prompt, while higher effort used more tokens, time, and cost. ### Source excerpt Today is Claude Fable (and Mythos) 5.1 day. Anthropic say that Fable 5.1 "sets a new standard for coding, knowledge work, and long-running problem-solving tasks". Their announcement spends a notable amount of time on scientific research, boasting of a 52.6% score on the brand new Terminal-Bench-Science 0.1 benchmark (first announced on August 27th), up from 24.7% for Fable 5, 29.0% for Opus 5 and 22.4% for GPT-5.6 Sol. Other benchmarks show slightly improved scores, but none as impressive as the Science one. But how well can it pelican? Back in July I wrote about how I was losing faith in the pelican benchmark - its connection to how good the models were at other tasks didn't seem to hold as strongly as it did back in 2025. The most interesting insights I get from it now are comparisons within model families, and particularly comparisons for the same prompt at different reasoning effort levels. Fable 5.1 has five reasoning levels: low, medium, high, xhigh, max - and no option to turn off reasoning entirely. I fixed an issue in llm-anthropic which caused reasoning traces not to be correctly recorded, then ran some prompts. Here's the full set of pelicans for all of the reasoning levels, each with the full reasoning transcript. I'll replicate them here: Low and medium, both without reasoning? Next, a bit of a mystery. This is what I got for effort low: The transcript doesn't show any summarized reasoning tokens, and the output token count is 1,998. With Claude that output token count includes reasoning tokens. It took 23.8 seconds and cost 10.017 cents. I bumped that up to medium and got this: Weirdly, that one also shows no reasoning text and used 1,977 output tokens - 21 tokens less than low. It took 23 seconds and cost 9.912 cents. So for this particular prompt ("Generate an SVG of a pelican riding a bicycle") Fable 5.1 appeared to skip reasoning entirely at both low and medium settings. High Here's high - 29.6 seconds, 2,612 output tokens, 13.087 cents: This one d ## Qwen 3.8 27B is excellent, but it defaults to wildly overthinking things DevFeed: [Qwen 3.8 27B is excellent, but it defaults to wildly overthinking things]() Original publisher: [Read original article]() Author: Simon Willison Published: 2026-08-16T22:00:39Z Content type: opinion Language: en Sources: [Simon Willison]() Topics: [qwen](), [Large Language Model](), [Hardware](), [DGX Spark](), [NVIDIA DGX](), [SVG]() Tags: [ai](), [ai-2-235](), [ai-in-china](), [ai-in-china-108](), [coding-agents](), [coding-agents-248](), [cost](), [dgx-spark](), [generative-ai](), [generative-ai-1-981](), [llama-cpp](), [llama-cpp-29](), [llm](), [llm-reasoning](), [llm-reasoning-103](), [llm-release](), [llm-release-231](), [llms](), [llms-1-947](), [lm-studio](), [lm-studio-23](), [local-llms](), [local-llms-164](), [nvidia-dgx](), [nvidia-spark](), [nvidia-spark-6](), [pelican-riding-a-bicycle](), [pelican-riding-a-bicycle-142](), [pi](), [pi-6](), [qwen](), [qwen-61](), [reasoning](), [speed](), [svg]() ### AI overview Simon Willison evaluates Qwen 3.8 27B, a vision-capable 27-billion-parameter LLM that can run locally on suitable hardware. He finds that its default xhigh reasoning setting consumes substantial context and time, while adjusting the reasoning effort and increasing the context limit improves practicality. He also reports strong results generating an SVG locally. ### Source excerpt Friday's big release was Qwen 3.8 27B, an Apache 2 licensed 27B parameter vision-capable LLM from Alibaba's Qwen research lab. I've been looking forward to this one: 27B is an excellent size for running a model on a reasonably specced laptop, and its predecessor Qwen 3.6 27B was impressive. Qwen's self-reported benchmarks for this model are eye-opening. They show a boost from both Qwen 3.6 27B and the closed-weight Qwen 3.7-Plus, which was one of Qwen's strongest models of any size as recently as May this year. It will be interesting to hear what independent benchmarks have to say about the model. I've been running the model on two different machines: my 128GB M5 Max MacBook Pro, and an NVIDIA DGX Spark. On both machines I'm running LM Studio and their 17GB Q4_K_M quantized build. I also tried using llama-server directly on the Spark. The default of extra high results in spectacular over-thinking Qwen's documentation describes the model as defaulting to xhigh for the reasoning effort, and the LM Studio GGUF I've been trying preserves that default: Qwen3.8 comes with official support for reasoning_effort, which can be used to adjust reasoning depth and control cost: xhigh (default): for complex tasks demanding thorough analysis medium: balancing accuracy and speed low: efficient reasoning optimizing for speed and cost This is a hilarious default. It's absolutely not a good way to run the model, especially on consumer hardware. I've been finding the results extremely entertaining. I quickly ran into problems with LM Studio's default context limit of 8,192 tokens - Qwen was using them all up thinking about even the most mundane of problems. I loaded the model with the full 262,144 maximum context length and that problem went away. Here's the pelican riding a bicycle SVG I got from my first attempt with that increased context length. It took 21 minutes to generate, using 22,276 reasoning tokens to produce 3,223 tokens of output. You can read the reasoning trace here. Th ## New release of LLM adds support for reasoning traces, OpenAI Responses, server-side tools, and smarter logging DevFeed: [New release of LLM adds support for reasoning traces, OpenAI Responses, server-side tools, and smarter logging]() Original publisher: [Read original article]() Author: Simon Willison Published: 2026-08-04T23:58:24Z Content type: release Language: en Sources: [Simon Willison]() Topics: [Tool](), [Command-line interface](), [API](), [Large Language Model](), [SQLite](), [Logging](), [OpenAI](), [Claude](), [Model Context Protocol](), [gemma4]() Tags: [ai](), [ai-2-235](), [anthropic](), [anthropic-336](), [cli](), [gemma](), [generative-ai](), [generative-ai-1-981](), [llm](), [llm-631](), [llm-reasoning](), [llm-reasoning-103](), [llm-tool-use](), [llm-tool-use-75](), [llms](), [llms-1-947](), [logging](), [mcp](), [model-context-protocol](), [model-context-protocol-35](), [openai](), [openai-463](), [projects](), [projects-553](), [release](), [releases](), [releases-31](), [responses](), [server](), [sqlite]() ### AI overview LLM 0.32 adds visible reasoning traces, server-side provider tools, redesigned content-addressable SQLite logs, support for the GPT-5.6 model family, and features enabled by the OpenAI Responses API. The release also updates the llm-anthropic plugin and adds Python API changes. ### Source excerpt I released LLM 0.32 this morning, the most significant new version of LLM since the initial launch of the project. The new version includes support for visible reasoning traces, server-side provider tools, redesigned content-addressable SQLite logs, new models, and new features enabled by the OpenAI Responses API. I also released a new version of the llm-anthropic plugin with substantial updates of its own. Headline features for LLM CLI users Running LLM against reasoning models now displays their reasoning traces to standard error, so you can see what they are "thinking" without that information being included in the standard output that you might pipe to another tool. Add -R/--hide-reasoning to turn this off. LLM includes support out-of-the-box for the GPT-5.6 model family, and the new default model used with llm "prompt" is now the inexpensive but capable GPT-5.6 Luna. LLM calls can now use server-side tools from various providers. OpenAI provide a code execution environment as a server-side tool; LLM can now run prompts that benefit from that like so: llm --tool CodeInterpreter 'Show current python and SQLite versions' OpenAI also gets a WebSearch tool. The llm-anthropic plugin adds WebSearch, WebFetch, CodeExecution, and AnthropicMCP, which looks like this: llm -m claude-sonnet-5 -T 'AnthropicMCP("https://datasette.simonwillison.net/-/mcp")' \ 'how many rows in the blog_blogmark table?' That causes Anthropic to execute MCP calls against my new datasette-mcp plugin as part of a single request/response interaction with their API. The new llm openai endpoint command provides a tool for executing prompts against any OpenAI compatible endpoint as a one-liner. These aren't logged, which makes this a handy tool for running one-off prompts against anything that speaks the lingua franca of the LLM API world. Here's how I use that to run prompts against Gemma 4 12B running in my localhost LM Studio API, via uvx (no LLM installation required) and mixing in the llm-tools-q ## LLM reasoning and agentic safety at ICML 2026 DevFeed: [LLM reasoning and agentic safety at ICML 2026]() Original publisher: [Read original article]() Author: Capital One Tech Published: 2026-07-02T14:14:40Z Content type: article Language: en Sources: [Capital One Tech]() Topics: [Large Language Model](), [Fine-tuning](), [Trustworthy AI](), [Artificial Intelligence]() Tags: [2026](), [ai](), [ai-research](), [fine-tuning](), [icml](), [icml-2026](), [llm](), [llm-reasoning](), [reasoning](), [research](), [safety](), [science](), [trustworthy-ai]() ### AI overview Capital One presents research for ICML 2026 on critique-guided distillation for robust LLM reasoning and on safety risks in multi-turn tool-using agents. The article says its Critique-Guided Distillation framework trains models to refine flawed responses using teacher critiques, and reports higher mathematical-reasoning benchmark performance than critique fine-tuning and standard distillation, including a 7% average improvement and gains of up to 15.0% on AMC23 and 12.2% on MATH-500. ### Source excerpt Explore our latest research in critique-guided distillation and multi-turn agent uncertainty in Seoul.Explore our latest research in critique-guided distillation and multi-turn agent uncertainty in Seoul. Capital One technologists are excited to participate in the 43rd International Conference on Machine Learning (ICML) taking place at the COEX Convention & Exhibition Center in Seoul, South Korea, July 6-11, 2026. As a premier global venue for machine learning research, ICML provides an essential forum for exploring foundational advancements, algorithmic innovations and cutting-edge deep learning systems. Capital One is excited to share advancements in large language model (LLM) scaling efficiencies, multi-turn tool-using agent safety and the development of robust, trustworthy AI frameworks. This work delivers the underlying engineering and algorithmic improvements crucial for deploying the next generation of safe financial technologies. Main conference research: Robust reasoning and agentic risk The following research, accepted to the ICML Main Conference, pushes the boundaries of how models self-correct, how trajectory-level risks can be proactively flagged, and how multi-turn agent interactions maintain reliable execution. This section features work led by Capital One researchers alongside deep collaborations with academic partners. Critique-Guided Distillation for Robust Reasoning via Refinement Capital One Authors: Berkcan Kapusuzoglu, Supriyo Chakraborty, Michael Lee, Sambit Sahu Supervised fine-tuning with expert demonstrations often produces models that imitate outputs without internalizing the reasoning processes needed for robust generalization. While critique-based approaches show promise, training models to generate critiques directly, such as Critique Fine-Tuning (CFT), can lead to output-format drift and degradation of general capabilities. We propose Critique-Guided Distillation (CGD), a training framework that decouples critique consumption from crit