BFSI-Bench

Key Findings

• GPT-6 Astra (67.5%), GPT-6.1 Sol (64%), and Grok 4.7 (61%) lead the benchmark.
• Questions with a source published in 2026 are much harder than questions with an earlier publication date: the models are correct on 39% of the 2026-dated questions and 82% of the earlier ones.
• Models search more on the questions they get wrong. A run with three or four searches is correct 66% of the time, while a run with nine or more is correct only 37% of the time.
• Search often returns a private website instead of the official page. TaxGuru, a private tax site, appears in 40% of searches, while the official income-tax sites appear in 15%.
• Most wrong answers repeat an older rule, either because the model searched for that rule or because it saw the new rule and still gave the old one.

Methodology

The Task

There are 200 questions. Each gold answer is a set of facts, easy to check, such as a yes or no, a form, or a rate. We take each fact from an official document on a government site such as the Reserve Bank of India, SEBI, or IRDAI.

Evaluation and Grading

Each model answers in Inspect’s ReAct agent, the same agent the BrowseComp eval uses. The only search tool is Exa, so every model gets the same search. The model can search, read the result, and search again. The run stops after 24 messages. We use GLM-5.3-Flash as a reference-based LLM judge to evaluate the answers. It checks the answer against the gold facts. An answer is correct only when it states every vital fact. Leaving a non-vital fact out is allowed. Contradicting a vital fact makes the answer incorrect, and a missing vital fact is marked not attempted.

Results

GPT-6 Astra leads at 67.5%, then GPT-6.1 Sol at 64%, then Grok 4.7 at 61%. Muse Spark 1.3 is close, at 60.5%, and much cheaper at about $0.02 a question.

A run with three or four searches is correct on 66% of questions. A run with nine or more searches is correct on 37% of questions.

104 questions have a main source published in 2026, and 61 have an earlier publication date. Accuracy is 39% on the 2026 questions and 82% on the earlier ones. For every model, accuracy on the 2026 questions is 33 to 50 percentage points lower.

Error analysis

About 46% of answers were not correct. About 68% of the wrong answers involve an older rule, or a current rule used the wrong way. In some the model searched for the old rule, in some it saw the current rule and still answered wrong, and in some the search returned only the old rule.

The other wrong answers fall into two modes. In 15% the model does not search, and answers from training data that predates many of the 2026 rules. Claude Opus 5.5 accounts for about two thirds of these. In 17% the run stops after 24 messages, before the model gives a correct answer.

• Searched for the old rule (30%)
• Saw the current rule, still answered wrong (23%)
• Search found only the old rule (15%)
• Answered from memory (15%)
• Ran out of turns (17%)

0%20%40%60%

GPT-6
Astra 32.5% GPT-6.1
Sol 36% Grok
4.7 39% Muse
Spark 1.3 39.5% Opus
5.5 46% Qwen
3.8 Max 52% DeepSeek
V4.1 Flash 54% Gemini
3.8 Flash 57.5% Mistral
Large 4 59%

• GPT-6 Astra 32.5%
• GPT-6.1 Sol 36%
• Grok 4.7 39%
• Muse Spark 1.3 39.5%
• Opus 5.5 46%
• Qwen 3.8 Max 52%
• DeepSeek V4.1 Flash 54%
• Gemini 3.8 Flash 57.5%
• Mistral Large 4 59%

Below, we show samples of how runs went wrong.

Searched for the old rule Saw the current rule, still answered wrong Search found only the old rule Answered from memory

Searched for the old rule

30% of wrong answers

The question does not name an old form, section, or rate, but the model adds one to its search, and the results then describe that older rule.

We asked GPT-6.1 SolWe run a jewellery shop in India. A walk-in customer wants to buy a ₹3.2 lakh necklace today and has no PAN. What do we get her to sign?

1. Current ruleForm No. 97, under the Income-tax Rules, 2026, in force from 1 April 2026.
2. Search

   income tax rule 114B sale goods services exceeding two lakh form 60 jewellery no PAN

   Search result

   If a person does not have a PAN, they must furnish a declaration in Form 60.

3. AnswerGet her to complete and sign Form 60, the prescribed declaration for a person without PAN.

View complete trajectory

Contact

If you have a question, want to license the benchmark, or evaluate your model, contact us at hello@ground-truth.in.

@dataset{groundtruth2026bfsibench,
  title={BFSI-Bench: Fact-Seeking Eval for Indian BFSI},
  author={GroundTruth},
  year={2026},
  url={https://www.ground-truth.in/benchmarks/bfsi-bench},
  license={CC-BY-4.0}
}