> ## Content Index > Fetch the complete content index at: https://thecoverwire.com/llms.txt > Use this file to discover other available public pages before exploring further. # FCA warns firms against purely technical explanations of AI decisions - URL: https://thecoverwire.com/interpretability-and-explaining-ai-pricing-decisions/ - Published: 2026-11-09T07:30:00.000Z - Updated: 2026-11-09T07:30:00.000Z - Description: Research says AI models may be easier to read than thought. The FCA wants explanations of AI decisions that show customer outcomes, not mechanics. - Author: CoverWire - Tags: Analysis, AI, #desk-risk **Analysis** | By CoverWire, created using AI Insurers that use AI to price or underwrite may soon have better tools to look inside their models. The Financial Conduct Authority has made clear that this alone will not be enough. It lists "taking a purely technical approach to explaining their model's actions" as an area for improvement. CoverWire's analysis is that interpretability research helps with one part of the job, and the harder part is showing what the model did to customers. ## Key points - AI research firm Goodfire says its own preliminary audits suggest about 90% of parts of one open AI model have one main job, against about 2% of neurons judged readable in a 2023 study on a different measure. - Goodfire calls this a working hypothesis. It is not peer reviewed, and the figure depends on how thresholds are set. - The FCA wants firms to be able to explain AI decisions, and warns against explanations that only describe how a model works. - Under the EU AI Act, AI used for pricing life and health insurance for individuals is high risk, with those rules now due from 2 December 2027. ## What the research claims Goodfire published an [essay](https://already-clean.goodfire.com/?ref=thecoverwire.com) in October arguing that neural networks are more orderly inside than many researchers assumed. In preliminary audits of the open model Qwen3-32B, it estimates that roughly 90% of what it calls neuron poles have one main job, about 7% have two related jobs, and only 2% to 4% are not yet understood. The essay says a 2023 study found about 1.7% of neurons readable on its most generous measure. These are Goodfire's own claims. The essay describes itself as informal and a working hypothesis, and says the share changes with how thresholds are set. It has not been independently replicated. A plain account of how interpretability research works, and of the gap between what is observed, inferred and tested, was set out by Nate B. Jones in [a post explaining mechanistic interpretability](https://natesnewsletter.substack.com/p/mechanistic-interpretability-explained). ## What the regulator wants The FCA's [good practice examples on AI](https://www.fca.org.uk/publications/good-poor-practice/ai-financial-services?ref=thecoverwire.com), published on 19 October, say firms should use proportionate controls "to ensure that decisions can be explained". The page links this to its Principles for Businesses and the [Consumer Duty](https://www.fca.org.uk/firms/consumer-duty?ref=thecoverwire.com). Its warning about technical explanations is specific. It describes firms that "focused only on how a model works, rather than whether they could show good outcomes, spot poor outcomes, understand the likely causes and support challenge or redress." In the EU, [Annex III of the AI Act](https://eur-lex.europa.eu/eli/reg/2024/1689/oj?ref=thecoverwire.com) lists AI used for risk assessment and pricing of individuals in life and health insurance as high risk. The [AI Omnibus](https://digital-strategy.ec.europa.eu/en/news/ai-omnibus-enters-force?ref=thecoverwire.com), in force since 27 July, moved the start date for those rules to 2 December 2027. ## Why this matters for pricing This section is CoverWire's interpretation. If a model's parts really are mostly readable, a pricing team could in principle check whether a model leans on a proxy for a protected characteristic. That would help with the "understand the likely causes" part of the FCA's test. It would not answer the rest. Explaining that a component tracks postcode density does not show whether a group of customers ended up paying more, or give a customer a reason they can challenge. Those are outcome questions, and they are answered with pricing data and testing, not by inspecting the model. ## The other explanation The research may matter less than it appears. Pricing models that are not large neural networks may already be explainable with existing tools. Goodfire's figures come from one model and are preliminary. On that view, pricing teams should treat this as a research signal, not a change in what they can show a regulator today. Related coverage: [BoE and FCA survey finds 93% of insurers use AI](https://thecoverwire.com/boe-fca-survey-93-percent-insurers-use-ai/). ## Sources - [Goodfire, "Neural Networks Are Already Clean", October 2026](https://already-clean.goodfire.com/?ref=thecoverwire.com) - [FCA, AI in financial services: good practice and areas for improvement, 19 October 2026](https://www.fca.org.uk/publications/good-poor-practice/ai-financial-services?ref=thecoverwire.com) - [FCA, Consumer Duty](https://www.fca.org.uk/firms/consumer-duty?ref=thecoverwire.com) - [Regulation (EU) 2024/1689, the AI Act, Annex III (EUR-Lex)](https://eur-lex.europa.eu/eli/reg/2024/1689/oj?ref=thecoverwire.com) - [European Commission, AI Omnibus enters into force](https://digital-strategy.ec.europa.eu/en/news/ai-omnibus-enters-force?ref=thecoverwire.com) - Idea credit: [Nate B. Jones, "What Is Mechanistic Interpretability?", 22 October 2026](https://natesnewsletter.substack.com/p/mechanistic-interpretability-explained)