Cultural Riddles is a multilingual benchmark of 6,191 original riddles written by native speakers from 62 communities. It tests whether language models can solve riddles that depend on lived experience, local associations, traditions, and wordplay. This repository holds the data, the benchmark runner, and the code that reproduces every figure, table, and number in the paper CulturalRiddles: A Multicultural Benchmark of Riddles.
The released snapshot records 170,520 answers from 28 model configurations on 6,090 benchmarked riddles. Each answer is scored by rules, anonymous native-speaker labels, or two agreeing LLM judges.
The dataset, its schemas, and its terms are available at these locations:
- Dataset: CohereLabsCommunity/CulturalRiddles on Hugging Face
- Play: try the riddles yourself and compare your answers with AI models
- Communities: the communities already in the benchmark
- Results dashboard: Cultural Riddles on Hugging Face Spaces
- Data contract: schemas, score meanings, and snapshot boundaries
- Licenses: code under Apache-2.0; data under the separate data terms
Is your community missing? You can add it in a few steps, with no GitHub or coding needed. Everything is on one page: Add your community's riddles.
You need uv and nothing else. It installs the pinned Python 3.12 environment for you.
git clone https://github.com/Cohere-Labs-Community/CulturalRiddles.git
cd CulturalRiddles
uv sync --frozenThat's the whole setup. Run every command from this folder. Everything below works offline and needs no API key.
Choose what you want to do. The following table links each goal to a step-by-step walkthrough in the getting started guide.
| Your goal | Start with | You get |
|---|---|---|
| Add your community's riddles | The one-page guide; no setup or coding needed | Your riddles in the dataset, with credit for your community |
| Benchmark a model | uv run --frozen riddles-framework --config benchmark/configs/dummy.yaml --limit 5 |
Raw answers under data/runs/, ready to score |
| Reproduce the paper | uv run --frozen python -m analysis.run_all |
Figures, LaTeX tables, number macros, and JSON claims |
| Use the data | pd.read_json("data/canonical/riddles.jsonl", lines=True) |
One row per riddle |
Reproducing the paper doesn't need R, because analysis.run_all replays the released summaries of the statistical models. Refitting those models with analysis.difficulty_decomposition is optional and slow, and it requires R with the lme4 and data.table packages. For more information, see Refit all statistical models.
Only paid model or judge calls need credentials. Create a local .env without overwriting an existing one, fill in only the keys you need, and never commit it:
cp -n .env.template .envJudges call Gemini and DeepSeek directly, so they need GEMINI_API_KEY and DEEPSEEK_API_KEY. Solver providers come from the model configuration. See credentials and provider routing.
The following guides cover each area in depth:
| Guide | Read it to |
|---|---|
| Add your community's riddles | Write riddles for your community and send them, with or without GitHub |
| Contributing with git | Check a workbook, open a pull request, review imports, and propose corrections |
| Submission tools | Learn the validation rules and the maintainer import workflow |
| Benchmark guide | Run and score models, configure providers, and run judges |
| Reproduction guide | Verify results, refit the statistical models, and build a release archive |
| Analysis guide | Trace each figure, table, and claim to its code |
| Human evaluation guide | Validate additional annotation workbooks |
| Dashboard guide | Build the optional results dashboard |
| AGENTS.md | Follow the maintenance boundaries and verification checks |
The repository has this layout:
benchmark/ Model adapters, runner, judge runner, configurations, and guides
submissions/ Anonymous intake checks, approved imports, corrections, and publishing
human_eval/ Validation code for anonymous annotation workbooks
analysis/ Scoring, aggregate analysis, figures, tables, and claims
scripts/ Data validation and self-contained release verification
dashboard/ Standalone renderer for exported aggregate JSON
docs/ Getting started and reproduction guides
data/
canonical/ Approved riddles, responses, human labels, judge evidence, and aggregates
reference/ External reference inputs
derived/ Rebuilt analysis outputs and retained R model summaries
manifests/ Approved input schemas, keys, counts, and hashes
submissions/ Blank intake template; your local workbooks stay ignored by Git
runs/ Your local model and judge runs (ignored by Git)
figures/ Rendered paper and preprint figures
Install the commit hooks, and then run the checks from the repository root:
uv run --frozen pre-commit install --install-hooks
uv run --frozen pytest
uv run --frozen ruff check .
uv run --frozen ruff format --check .To build and verify a self-contained archive after you regenerate the outputs, see Verify a self-contained release.
If you use Cultural Riddles, please cite the paper. The BibTeX is also in CITATION.bib.
@misc{nguyen2026culturalriddles,
title = {CulturalRiddles: A Multicultural Benchmark of Riddles},
author = {Nguyen, My Chiffon
and Hu, Tianyi
and Anbalagan, Vinod
and Gagnier, Henry
and Sadeghi, Pouya
and Abdou, Farah
and Pricope, Tidor-Vlad
and Oliveira, Diego MMS
and D'Arcy, Jocelyn K
and Smith, Madeline
and Ajayi, Oluwaseun A
and Al Ghussin, Yusser
and Al-Khalili, Zena
and Altomare, Micol
and Amir, Shayan
and Aziz, Ahanaf
and Chen, Bryan
and Dhar, Ruchira
and El Mekki, Abdellah
and Elframawy, Rawan Mohammed
and Eppa, Akhil
and Fernandez, Silvia
and Gimei, Alex
and Hariharan, Arya
and Hoang, Christian
and Kadiyala, RamMohanRao
and Kozachek, Diana
and Limkonchotiwat, Peerat
and Maity, Ankita
and Mo, Di
and Munot, Ayush
and Naebzadeh, Aylin
and Nikitina, Anastasiia
and Pisarevskaya, Dina
and Poudel, Divas
and Purbey, Jebish
and Rabius Sany Apu, Md
and Ramadhan, Ahmad Mufli
and Rizova, Ralitza Nikolova
and Serrano-Miranda, Diogo M.
and Silimkhan, Vivek Gopalkrishna
and Solis Navarro, Luis Fernando
and Sun, Kaiser
and Suneesh, Arya
and Sunkara, Manaswitha
and Sunkara, Tejaswi
and Swindell, Jonathan E
and Yukhymenko, Hanna
and Zheng, Jingyi
and Zheng, Weihua
and Adenuga, Esther
and Afanasev, Ilia
and Agarwal, Amit
and Aissa, Wafa
and Alazraki, Lisa
and Altomare, Lauren C
and Amuok, Priscilla Amondi
and Anand, Emile
and Anbalagan, Vivek
and Assis, Gabriel
and Avalos, José Matías
and Avila, Silvia Lorena
and Baannou, Zakaria
and Bafna, Niyati
and Bagla, Yash
and Bahrami, Sarah
and Balcha, Bontu F
and Bannihatti Kumar, Vanya
and Baucells, Irene
and Begmuratov, Abdumutalib
and Berk, Ceren
and Bhatia, Naman
and Bhattacharyya, Souvik
and Binte Md Johan, Nabilah
and Blanco Ferreiro, Isabel
and Bochini, Felipe
and Bopitiya, Sayuru
and Borah, Angana
and Bose, Tulika
and Bugaiova, Olena
and Carvalho, Juliana
and Chen, Nancy F
and Chen, Stephy
and Choenni, Rochelle
and Chundru, Jayanth Krishna
and Cossio, Leandro A
and Damnjanović, Nebojša
and de Gibert, Ona
and De Santis, Marco
and Delgado, Pedro
and Dholé, Kaustubh
and El Assadi, Adnan
and El-Sayed, Wesam
and Elframawy, Nadine M
and Essarhi, Oumayma
and Etori, Naome A
and Fernandes, Reuben Chagas
and Gade, Saanidhi Pravin
and García Gilabert, Javier
and González-Escapa, Álvaro
and Hu, Bin
and Huang, Kaiyu
and Ilasariya, Fenal Ashokbhai
and Issam, Abderrahmane
and Jokić, Danka
and Khalilov, Ulugbek
and Krzemiński, Dominik
and Lee, Roy Ka-Wei
and Li, Haichang
and Liu, Chen Cecilia
and Liu, Zhengyuan
and Lothritz, Cedric
and Mehrafarin, Houman
and Miranda-Escalada, Antonio
and Mo, Fengran
and Mofakhami, Mehrnaz
and Mohammed, Wafaa
and Moreno-Vera, Felipe
and Movva, Prahitha
and Nayem, Jannatul
and Ngo, Van
and Olusanya, Joy
and Pàmies, Marc
and Pandey, Avinash Kumar
and Paniv, Yurii
and Patel, Hitesh Laxmichand
and Philippy, Fred
and Poritski, Vladislav
and Poudel, Dikshya
and Radoła, Joanna
and Rajaee, Sara
and Rajwal, Swati
and Rane, Prerana
and Ravi Shulthan Habibi, Muhammad
and Resck, Lucas
and Roy Dipta, Shubhashis
and Sahu, Ananya
and Sánchez, Eduardo
and Sánchez Clemente, Borja
and Sant, Aleix
and Sarwar, Nobin
and Shihab, Istiak
and Singaraju, Santhoshini Pranathi
and Smith, Jane
and Stap, David
and Su, Emily Z
and Suresh, Ananth K
and Swami, Sajag
and Tan, Ruiyang
and Tasnim, Nazia
and Tounoussi, Khalid
and Triamamornwooth, Patteera
and Umar, Muhammad Saad
and Venkitachalam, Shankar
and Volchek, Oksana
and Wang, Lingbo
and Yadeta, Debela Desalegn
and Zawar, Rushikesh
and Zhang, Jinghan
and Zhang, Mike
and Fadaee, Marzieh
and Arora, Akhil
and Boczka, Károly
and Kreutzer, Julia},
year = {2026},
eprint = {2610.29968},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2610.29968},
}Code uses Apache-2.0. The dataset has separate terms in data/LICENSE: attribution is required, and you must not use the data to train, fine-tune, or distill models, or to seed synthetic data. The data terms aren't unmodified CC-BY-4.0.
