Skip to content

About

Official codebase for "CulturalRiddles: A multicultural benchmark of riddles"

Topics

Resources

Contributing

Stars

15 stars

Watchers

0 watching

Forks

Repository files navigation

Cultural Riddles

arXiv Dataset on Hugging Face Results dashboard on Hugging Face Play the riddles Code: Apache-2.0 Data terms

Cultural Riddles is a multilingual benchmark of 6,191 original riddles written by native speakers from 62 communities. It tests whether language models can solve riddles that depend on lived experience, local associations, traditions, and wordplay. This repository holds the data, the benchmark runner, and the code that reproduces every figure, table, and number in the paper CulturalRiddles: A Multicultural Benchmark of Riddles.

The released snapshot records 170,520 answers from 28 model configurations on 6,090 benchmarked riddles. Each answer is scored by rules, anonymous native-speaker labels, or two agreeing LLM judges.

Overview of the benchmark. Panel A shows example riddles from Canada and Thailand and the topics they cover. Panel B shows how answers are scored: rules first, then native speakers, then an LLM jury. Panel C summarizes the results: accuracy falls with fewer language resources, and models more often hallucinate than admit that they don't know.

Links

The dataset, its schemas, and its terms are available at these locations:

Contribute riddles

Is your community missing? You can add it in a few steps, with no GitHub or coding needed. Everything is on one page: Add your community's riddles.

Get started

You need uv and nothing else. It installs the pinned Python 3.12 environment for you.

git clone https://github.com/Cohere-Labs-Community/CulturalRiddles.git
cd CulturalRiddles
uv sync --frozen

That's the whole setup. Run every command from this folder. Everything below works offline and needs no API key.

Choose what you want to do. The following table links each goal to a step-by-step walkthrough in the getting started guide.

Your goal Start with You get
Add your community's riddles The one-page guide; no setup or coding needed Your riddles in the dataset, with credit for your community
Benchmark a model uv run --frozen riddles-framework --config benchmark/configs/dummy.yaml --limit 5 Raw answers under data/runs/, ready to score
Reproduce the paper uv run --frozen python -m analysis.run_all Figures, LaTeX tables, number macros, and JSON claims
Use the data pd.read_json("data/canonical/riddles.jsonl", lines=True) One row per riddle

Reproducing the paper doesn't need R, because analysis.run_all replays the released summaries of the statistical models. Refitting those models with analysis.difficulty_decomposition is optional and slow, and it requires R with the lme4 and data.table packages. For more information, see Refit all statistical models.

Credentials for paid calls

Only paid model or judge calls need credentials. Create a local .env without overwriting an existing one, fill in only the keys you need, and never commit it:

cp -n .env.template .env

Judges call Gemini and DeepSeek directly, so they need GEMINI_API_KEY and DEEPSEEK_API_KEY. Solver providers come from the model configuration. See credentials and provider routing.

Guides

The following guides cover each area in depth:

Guide Read it to
Add your community's riddles Write riddles for your community and send them, with or without GitHub
Contributing with git Check a workbook, open a pull request, review imports, and propose corrections
Submission tools Learn the validation rules and the maintainer import workflow
Benchmark guide Run and score models, configure providers, and run judges
Reproduction guide Verify results, refit the statistical models, and build a release archive
Analysis guide Trace each figure, table, and claim to its code
Human evaluation guide Validate additional annotation workbooks
Dashboard guide Build the optional results dashboard
AGENTS.md Follow the maintenance boundaries and verification checks

Repository map

The repository has this layout:

benchmark/      Model adapters, runner, judge runner, configurations, and guides
submissions/    Anonymous intake checks, approved imports, corrections, and publishing
human_eval/     Validation code for anonymous annotation workbooks
analysis/       Scoring, aggregate analysis, figures, tables, and claims
scripts/        Data validation and self-contained release verification
dashboard/      Standalone renderer for exported aggregate JSON
docs/           Getting started and reproduction guides
data/
  canonical/    Approved riddles, responses, human labels, judge evidence, and aggregates
  reference/    External reference inputs
  derived/      Rebuilt analysis outputs and retained R model summaries
  manifests/    Approved input schemas, keys, counts, and hashes
  submissions/  Blank intake template; your local workbooks stay ignored by Git
  runs/         Your local model and judge runs (ignored by Git)
figures/        Rendered paper and preprint figures

Develop

Install the commit hooks, and then run the checks from the repository root:

uv run --frozen pre-commit install --install-hooks
uv run --frozen pytest
uv run --frozen ruff check .
uv run --frozen ruff format --check .

To build and verify a self-contained archive after you regenerate the outputs, see Verify a self-contained release.

Citation

If you use Cultural Riddles, please cite the paper. The BibTeX is also in CITATION.bib.

@misc{nguyen2026culturalriddles,
  title = {CulturalRiddles: A Multicultural Benchmark of Riddles},
  author = {Nguyen, My Chiffon
            and Hu, Tianyi
            and Anbalagan, Vinod
            and Gagnier, Henry
            and Sadeghi, Pouya
            and Abdou, Farah
            and Pricope, Tidor-Vlad
            and Oliveira, Diego MMS
            and D'Arcy, Jocelyn K
            and Smith, Madeline
            and Ajayi, Oluwaseun A
            and Al Ghussin, Yusser
            and Al-Khalili, Zena
            and Altomare, Micol
            and Amir, Shayan
            and Aziz, Ahanaf
            and Chen, Bryan
            and Dhar, Ruchira
            and El Mekki, Abdellah
            and Elframawy, Rawan Mohammed
            and Eppa, Akhil
            and Fernandez, Silvia
            and Gimei, Alex
            and Hariharan, Arya
            and Hoang, Christian
            and Kadiyala, RamMohanRao
            and Kozachek, Diana
            and Limkonchotiwat, Peerat
            and Maity, Ankita
            and Mo, Di
            and Munot, Ayush
            and Naebzadeh, Aylin
            and Nikitina, Anastasiia
            and Pisarevskaya, Dina
            and Poudel, Divas
            and Purbey, Jebish
            and Rabius Sany Apu, Md
            and Ramadhan, Ahmad Mufli
            and Rizova, Ralitza Nikolova
            and Serrano-Miranda, Diogo M.
            and Silimkhan, Vivek Gopalkrishna
            and Solis Navarro, Luis Fernando
            and Sun, Kaiser
            and Suneesh, Arya
            and Sunkara, Manaswitha
            and Sunkara, Tejaswi
            and Swindell, Jonathan E
            and Yukhymenko, Hanna
            and Zheng, Jingyi
            and Zheng, Weihua
            and Adenuga, Esther
            and Afanasev, Ilia
            and Agarwal, Amit
            and Aissa, Wafa
            and Alazraki, Lisa
            and Altomare, Lauren C
            and Amuok, Priscilla Amondi
            and Anand, Emile
            and Anbalagan, Vivek
            and Assis, Gabriel
            and Avalos, José Matías
            and Avila, Silvia Lorena
            and Baannou, Zakaria
            and Bafna, Niyati
            and Bagla, Yash
            and Bahrami, Sarah
            and Balcha, Bontu F
            and Bannihatti Kumar, Vanya
            and Baucells, Irene
            and Begmuratov, Abdumutalib
            and Berk, Ceren
            and Bhatia, Naman
            and Bhattacharyya, Souvik
            and Binte Md Johan, Nabilah
            and Blanco Ferreiro, Isabel
            and Bochini, Felipe
            and Bopitiya, Sayuru
            and Borah, Angana
            and Bose, Tulika
            and Bugaiova, Olena
            and Carvalho, Juliana
            and Chen, Nancy F
            and Chen, Stephy
            and Choenni, Rochelle
            and Chundru, Jayanth Krishna
            and Cossio, Leandro A
            and Damnjanović, Nebojša
            and de Gibert, Ona
            and De Santis, Marco
            and Delgado, Pedro
            and Dholé, Kaustubh
            and El Assadi, Adnan
            and El-Sayed, Wesam
            and Elframawy, Nadine M
            and Essarhi, Oumayma
            and Etori, Naome A
            and Fernandes, Reuben Chagas
            and Gade, Saanidhi Pravin
            and García Gilabert, Javier
            and González-Escapa, Álvaro
            and Hu, Bin
            and Huang, Kaiyu
            and Ilasariya, Fenal Ashokbhai
            and Issam, Abderrahmane
            and Jokić, Danka
            and Khalilov, Ulugbek
            and Krzemiński, Dominik
            and Lee, Roy Ka-Wei
            and Li, Haichang
            and Liu, Chen Cecilia
            and Liu, Zhengyuan
            and Lothritz, Cedric
            and Mehrafarin, Houman
            and Miranda-Escalada, Antonio
            and Mo, Fengran
            and Mofakhami, Mehrnaz
            and Mohammed, Wafaa
            and Moreno-Vera, Felipe
            and Movva, Prahitha
            and Nayem, Jannatul
            and Ngo, Van
            and Olusanya, Joy
            and Pàmies, Marc
            and Pandey, Avinash Kumar
            and Paniv, Yurii
            and Patel, Hitesh Laxmichand
            and Philippy, Fred
            and Poritski, Vladislav
            and Poudel, Dikshya
            and Radoła, Joanna
            and Rajaee, Sara
            and Rajwal, Swati
            and Rane, Prerana
            and Ravi Shulthan Habibi, Muhammad
            and Resck, Lucas
            and Roy Dipta, Shubhashis
            and Sahu, Ananya
            and Sánchez, Eduardo
            and Sánchez Clemente, Borja
            and Sant, Aleix
            and Sarwar, Nobin
            and Shihab, Istiak
            and Singaraju, Santhoshini Pranathi
            and Smith, Jane
            and Stap, David
            and Su, Emily Z
            and Suresh, Ananth K
            and Swami, Sajag
            and Tan, Ruiyang
            and Tasnim, Nazia
            and Tounoussi, Khalid
            and Triamamornwooth, Patteera
            and Umar, Muhammad Saad
            and Venkitachalam, Shankar
            and Volchek, Oksana
            and Wang, Lingbo
            and Yadeta, Debela Desalegn
            and Zawar, Rushikesh
            and Zhang, Jinghan
            and Zhang, Mike
            and Fadaee, Marzieh
            and Arora, Akhil
            and Boczka, Károly
            and Kreutzer, Julia},
  year = {2026},
  eprint = {2610.29968},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  url = {https://arxiv.org/abs/2610.29968},
}

License and use

Code uses Apache-2.0. The dataset has separate terms in data/LICENSE: attribution is required, and you must not use the data to train, fine-tune, or distill models, or to seed synthetic data. The data terms aren't unmodified CC-BY-4.0.

About

Official codebase for "CulturalRiddles: A multicultural benchmark of riddles"

Topics

Resources

Contributing

Stars

15 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages