HalachaBench (@HalachaBench)

Introducing HalachaBench: we gave AI programs from the makers of ChatGPT, Claude, Gemini and others real semicha exam questions from Israel's Chief Rabbinate, in Hebrew, with no books and no internet. The best scored 80 out of 100. Many scored under 11. 🧵

1,142

195,069

HalachaBench · Nov 3, 2026 · 2:00 PM UTC

Which semicha paper is hardest for AI? Top 3 models: Chuppah v'Kiddushin. The next 5: Niddah. 7 of 8 did best on Mitzvot HaTeluyot BaAretz. Only 12 questions per paper, and some grading sheets may be stricter, so treat it as a hint. Purely an intellectual exercise.

Chart: mean score out of 100 per exam paper, in this order: Chuppah v'Kiddushin, Niddah, Mikvaot, Tefillah u'Brachot, Mitzvot HaTeluyot BaAretz. Claude Fable 5.1 73, 77, 82, 82, 85; GPT-6 Astra 62, 65, 77, 79, 84; GPT-6.1 Sol 60, 67, 78, 76, 83; Claude Opus 5.5 63, 60, 70, 75, 78; Gemini 3.8 Flash 60, 55, 66, 66, 74; GPT-6.1 Terra 24, 21, 34, 48, 45; GPT-6 Luna 22, 17, 40, 39, 44; Claude Sonnet 5.5 26, 16, 31, 23, 48. Provisional: one AI judge (GPT-6 Astra), AI-written grading sheets not yet reviewed by rabbis.

ALT Chart: mean score out of 100 per exam paper, in this order: Chuppah v'Kiddushin, Niddah, Mikvaot, Tefillah u'Brachot, Mitzvot HaTeluyot BaAretz. Claude Fable 5.1 73, 77, 82, 82, 85; GPT-6 Astra 62, 65, 77, 79, 84; GPT-6.1 Sol 60, 67, 78, 76, 83; Claude Opus 5.5 63, 60, 70, 75, 78; Gemini 3.8 Flash 60, 55, 66, 66, 74; GPT-6.1 Terra 24, 21, 34, 48, 45; GPT-6 Luna 22, 17, 40, 39, 44; Claude Sonnet 5.5 26, 16, 31, 23, 48. Provisional: one AI judge (GPT-6 Astra), AI-written grading sheets not yet reviewed by rabbis.

107

HalachaBench · Nov 3, 2026 · 1:30 PM UTC

Two older OpenAI models, as @GregHBurnham of Epoch AI suggested. GPT-5.6 Sol: 77.3, 3rd of 24, ahead of GPT-6.1 Sol (72.8). GPT-6 Sol: 48.7. Same harness, same effort setting, same questions. The newer model scored 29 points lower. @OpenAI @EpochAIResearch

Bar chart of HalachaBench scores for 24 AI models, 0 to 100 with 95% intervals. Top five: Claude Fable 5.1 79.7, Gemini 4 Argon 78.2, GPT-5.6 Sol 77.3, GPT-6 Astra 73.3, GPT-6.1 Sol 72.8. GPT-6 Sol is 8th at 48.7.

ALT Bar chart of HalachaBench scores for 24 AI models, 0 to 100 with 95% intervals. Top five: Claude Fable 5.1 79.7, Gemini 4 Argon 78.2, GPT-5.6 Sol 77.3, GPT-6 Astra 73.3, GPT-6.1 Sol 72.8. GPT-6 Sol is 8th at 48.7.

608

HalachaBench · Nov 3, 2026 · 1:32 PM UTC

Why? GPT-6 Sol didn't make more mistakes (it got 2% of checks wrong, 5.6 got 2.5%). It just said less: ~200 words vs ~310, less thinking, and it covered 48% of the sources and rulings a full answer needs vs 77%. 5.6 did better on 57 of 59 questions.

HalachaBench · Nov 3, 2026 · 1:33 PM UTC

It also gave a flat ruling where the answer depends on a fact the question leaves open 33 times, vs 9 for 5.6. Newer isn't always better. On a semicha exam, the model that writes out the conditions and the sources wins. github.com/halachabench/hala…

GitHub - HalachaBench/halacha-bench-results: HalachaBench early results: AI models on real semicha...

HalachaBench early results: AI models on real semicha exam questions (provisional) - HalachaBench/halacha-bench-results

github.com

HalachaBench · Nov 2, 2026 · 9:52 PM UTC

Update: 5 models added. Gemini 4 Argon: 78.2, 2nd of 22, statistically tied with Claude Fable 5.1. Grok 4.7: 29.9, 10th. Grok 4.3: 10.8. Claude Opus 4.8: 48.0. Claude Sonnet 4.6: 29.6. The checks behind the Grok result: teapawt.org/HalachaBench/status/21…

New on HalachaBench: Gemini 4 Argon and Grok. Gemini 4 Argon: 78.2 out of 100, 2nd of 22, statistically tied with Claude Fable 5.1 (79.7). Grok 4.7: 29.9, 10th. Grok 4.3: 10.8. (Poll answer: under 65.) @GoogleDeepMind @xai

253

HalachaBench · Nov 2, 2026 · 9:47 PM UTC

2,140

more replies

HalachaBench · Nov 2, 2026 · 9:47 PM UTC

Grok 4.7 thinks a lot but writes short answers (~210 words vs ~660 for Argon), and the grading sheets give points for each source and step. Details and all checks: github.com/halachabench/hala…

GitHub - HalachaBench/halacha-bench-results: HalachaBench early results: AI models on real semicha...

HalachaBench early results: AI models on real semicha exam questions (provisional) - HalachaBench/halacha-bench-results

github.com

199

HalachaBench · Nov 2, 2026 · 9:47 PM UTC

Also added: two earlier Claude models. Claude Opus 4.8: 48.0 (7th). Claude Sonnet 4.6: 29.6 (11th), level with Claude Sonnet 5.5 (28.7). @AnthropicAI

HalachaBench · Nov 2, 2026 · 3:10 PM UTC

@HalachaBenchNov 2

Quick guess: which semicha paper did AI find hardest? Answer tomorrow, with the chart.

12%Chuppah v'Kiddushin

39%Niddah

26%Mikvaot

22%Tefillah u'Brachot

76 votes • Final results

583

HalachaBench · Nov 2, 2026 · 1:31 AM UTC

@HalachaBenchNov 2

Grok is next on HalachaBench. The top score so far is 80 out of 100 (Claude Fable 5.1). Where will Grok land?

7%New #1 (above 80)

40%65 to 80

29%Under 65

24%Just show me

161 votes • Final results

1,202

David/Dovid Bashevkin · Nov 1, 2026 · 11:23 PM UTC

HalachaBench retweeted

David/Dovid Bashevkin

@DBashIdeasNov 1

AI takes rabbinical ordination exams… 👀

649

73,856

HalachaBench · Nov 1, 2026 · 11:38 PM UTC

Thank you to @SefariaProject: before the grading sheets were sealed, an AI checked their cited sources against Sefaria's library, about 2,000 lookups. Sources Sefaria doesn't carry are marked unverified.

5,724

HalachaBench · Nov 2, 2026 · 12:00 AM UTC

@HalachaBenchNov 2

Ask an AI the same semicha question three times and you can get three different answers. Keep only each model's weakest try on every question and scores drop 7 to 11 points. The order doesn't change. You'd want the same answer every time you ask.

Chart: score out of 100 for each AI, average of its three tries vs. its worst try on each question. Claude Fable 5.1 79.7 vs 70.4; GPT-6 Astra 73.3 vs 66.2; GPT-6.1 Sol 72.8 vs 63.9; Claude Opus 5.5 69.1 vs 60.0; Gemini 3.8 Flash 64.4 vs 54.3; GPT-6.1 Terra 34.6 vs 25.6; GPT-6 Luna 32.4 vs 21.8; Claude Sonnet 5.5 28.7 vs 18.3. Provisional: one AI judge (GPT-6 Astra), AI-written grading sheets not yet reviewed by rabbis.

ALT Chart: score out of 100 for each AI, average of its three tries vs. its worst try on each question. Claude Fable 5.1 79.7 vs 70.4; GPT-6 Astra 73.3 vs 66.2; GPT-6.1 Sol 72.8 vs 63.9; Claude Opus 5.5 69.1 vs 60.0; Gemini 3.8 Flash 64.4 vs 54.3; GPT-6.1 Terra 34.6 vs 25.6; GPT-6 Luna 32.4 vs 21.8; Claude Sonnet 5.5 28.7 vs 18.3. Provisional: one AI judge (GPT-6 Astra), AI-written grading sheets not yet reviewed by rabbis.

5,737

HalachaBench · Nov 1, 2026 · 9:28 PM UTC

הכירו את HalachaBench: נתנו למודלים של בינה מלאכותית (של יוצרי ChatGPT, Claude, Gemini ועוד) שאלות אמיתיות ממבחני ההסמכה של הרבנות הראשית. הם ענו בעברית, בלי ספרים ובלי אינטרנט. המודל הטוב ביותר קיבל 80 מתוך 100. רבים קיבלו פחות מ-11. 🧵

1,867

more replies

HalachaBench · Nov 1, 2026 · 9:28 PM UTC

חשוב להדגיש: זהו תרגיל אינטלקטואלי בלבד. איש אינו חושב שבינה מלאכותית יכולה לפסוק הלכה. התוצאות ראשוניות וזמניות: מודל אחד כתב את המחוונים ומודל אחר בדק את התשובות, ושניהם גם נבחנו בעצמם. אף רב עדיין לא עבר על המחוונים או על הציונים.

161

HalachaBench · Nov 1, 2026 · 9:28 PM UTC