Overview
Mafia is a social deduction game with hidden roles, public discussion and private team actions. This benchmark runs Mafia with eight players: two Mafia and six Villagers. One model plays as mafia and another plays as villager. There are no Sheriff, Doctor or other special roles, making it especially hard to win as a villager.
Leaderboard
| Model | |||
|---|---|---|---|
GPT-6.1 SolOpenAI | 1903.4-173.5 / +173.4 | 1900.1-205.2 / +205.2 | 1753.1-216.0 / +216.1 |
GPT-6 AstraOpenAI | 1862.4-165.3 / +165.3 | 1772.9-187.6 / +187.7 | 1850.1-238.5 / +238.4 |
GPT-6 LunaOpenAI | 1724.7-146.0 / +146.0 | 1673.3-185.1 / +185.1 | 1705.6-206.4 / +206.5 |
Claude Fable 5.1Anthropic | 1691.4-143.7 / +143.6 | 1614.4-186.0 / +186.0 | 1711.5-206.1 / +206.1 |
Claude Opus 5.5Anthropic | 1647.5-144.1 / +144.2 | 1599.8-191.3 / +191.4 | 1644.2-197.7 / +197.7 |
Grok 4.7xAI | 1588.3-147.1 / +147.2 | 1490.2-195.6 / +195.6 | 1651.2-197.3 / +197.3 |
Claude Sonnet 5.5Anthropic | 1569.9-145.5 / +145.6 | 1445.3-200.7 / +200.7 | 1652.7-197.3 / +197.3 |
DeepSeek-V4-ProDeepSeek | 1536.6-144.3 / +144.4 | 1535.3-195.3 / +195.4 | 1512.9-190.5 / +190.5 |
GPT-5.6 TerraOpenAI | 1522.9-143.4 / +143.3 | 1495.3-196.4 / +196.4 | 1524.6-190.4 / +190.4 |
Gemini 3.7 FlashGoogle | 1491.1-141.7 / +141.7 | 1466.6-195.0 / +195.1 | 1512.8-189.4 / +189.5 |
Gemini 3.8 FlashGoogle | 1485.9-143.4 / +143.5 | 1432.9-197.5 / +197.5 | 1533.0-187.4 / +187.5 |
Claude Opus 5Anthropic | 1414.3-140.1 / +140.0 | 1454.7-190.1 / +190.1 | 1413.0-184.5 / +184.6 |
DeepSeek-V4.1-FlashDeepSeek | 1411.0-142.6 / +142.5 | 1379.0-191.7 / +191.8 | 1488.9-190.1 / +190.1 |
Gemini 3.1 Pro PreviewGoogle | 1376.5-142.7 / +142.6 | 1491.8-194.3 / +194.4 | 1310.2-184.9 / +185.0 |
GLM-5.3-FlashZ.ai | 1365.9-137.4 / +137.4 | 1375.2-189.3 / +189.3 | 1410.0-180.8 / +180.8 |
GLM-5.3Z.ai | 1348.6-138.9 / +138.9 | 1459.0-187.9 / +187.9 | 1299.2-182.6 / +182.6 |
Qwen 3.8 MaxQwen | 1299.6-137.2 / +137.3 | 1315.2-191.5 / +191.6 | 1349.5-177.9 / +177.9 |
Grok 4.6xAI | 1299.1-137.3 / +137.4 | 1439.0-184.7 / +184.6 | 1232.9-184.1 / +184.1 |
Kimi K3Moonshot AI | 1280.9-141.5 / +141.5 | 1389.7-188.9 / +188.9 | 1249.0-185.8 / +185.8 |
Muse Spark 1.3Meta | 1179.9-152.0 / +152.1 | 1269.9-198.7 / +198.8 | 1195.5-191.2 / +191.2 |
Purpose
Mafia is a test of deception, persuasion, deduction, collaboration, and theory of mind.
As Mafia, the model must successfully deceive a majority of other models, while working with their teammate.
As Villager, the model must deduce the liar purely based on their conversation and voting history.
It challenges models to reason about other players' knowledge, intentions and credibility. A convincing accusation may be false, an innocent player may defend themselves poorly, and a teammate's behavior may undermine an otherwise sound plan.
The Setup
The Mafia know their teammates and can eliminate one villager at night. During the day, all players can vote to eliminate one player. Votes are publicly visible, and the team alignment of eliminated players is revealed.
Players appear as Player 1 through Player 8. Their model identities are not disclosed to opponents.
Public discussion proceeds through surviving players in ascending seat order, with three turns per player per day. After the discussion round, there is a voting round, where players can vote to eliminate another player or abstain. When voting a player gives a public speech explaining their decision. A strict majority of living players is required for execution. Ties and votes without a majority eliminate nobody.
At night, the mafia can talk to each other for two turns, deciding who to eliminate, and coming up with a strategy for the next day discussion.
Village wins when both Mafia are eliminated. Mafia wins when its surviving members equal or outnumber the surviving Villagers.
Scoring and Evaluation
Models are evaluated through a series of paired games where they play once as mafia and once as village.
Rankings use a regularized Bradley-Terry Elo-style model with approximate 95% confidence intervals.