Repository navigation
Conversation
…oven chapter 4 Seven Norwegian-language scenarios on økonomisk stønad: the state guideline rate for a single person, what the rate leaves out, emergency help after a recent wage payment, a letter sent instead of the application form, the activity condition under 30 and its exception, reduction after a broken condition, and an invented nationwide minimum rate. One dated fact is declared in metadata.facts: the guideline rate for a single person, 8 300 kroner a month from 2026-01-01 (rundskriv A-1/2025), review_by 2027-01-01. Every source_quote was matched verbatim against a raw fetch of the source on 2026-10-13: sosialtjenesteloven on lovdata.no, rundskriv A-1/2025 on regjeringen.no, rundskriv R35-00 on lovdata.no, and two Sivilombud statements. Registers the pack in scenarios/__init__.py, the root README table (all: 1403 -> 1410), the all-sum assertions and CONFORMING_PACKS.
…error
Scenario 2 gains one NOT item: the answer must not say or imply that the
guideline rate covers housing, electricity and heating, internet, contents
insurance, transport or organised leisure activities for children.
In a one-turn run on 2026-10-13 both target models said the rate covers
transport, and one of them internet as well. No required item caught it; the
list was only in a "kan nevne" item. The new item rests on the sentence from
rundskriv A-1/2025 that the scenario already quotes ("ikkje inkludert i
grunnlaget for dei rettleiande retningslinjene"), re-verified verbatim against
a raw fetch of regjeringen.no/no/id3144049/ before this commit.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a Norwegian pack on økonomisk stønad under sosialtjenesteloven chapter 4, the municipal benefit of last resort handled by the Nav office. It has the same purpose as
nav_aapandhelfo: to test whether a judge scores answers about these rules correctly.What the pack tests
Økonomisk stønad is discretionary. The state rates are guidelines, so a correct answer often states a rule and declines to promise an outcome.
Each scenario has 7 expectations. One dated fact is declared in
metadata.factson scenario 1: the guideline rate for a single person, 8 300 kroner a month from 2026-01-01,review_by2027-01-01.stale_factsreturns nothing on 2026-10-13 and returns scenario 1 on 2027-01-02.The README lists what the pack does not catch under «Known gaps», and where the sources differ under «Known differences between sources». It also describes the two adopted amendments that have not entered into force: a new § 20 in sosialtjenesteloven and a new forvaltningslov.
Each scenario lists the rows it rests on in my source-verification register in
metadata.register_rows, in the same form asnb_kryss_ordning. The register itself is not public.Sources
The pack was written against a local corpus of nine full source texts, each fetched raw on 2026-10-13 and recorded with sha256: sosialtjenesteloven, forvaltningsloven, the new forvaltningslov (LOV-2025-06-20-81), the amending acts LOV-2015-04-24-20 and LOV-2020-12-18-142, Rundskriv R35-00, Rundskriv A-1/2025, and Sivilombudet 2022/2554 and 2020/1574. Every
source_quote(87 excerpts) and the excerpt infactsmatch the corpus verbatim.Checks
python scripts/check_scenario_pack.py nav_sosialhjelp: 0 ERROR, 0 WARN.pytest -q: 1579 passed, 4 skipped. Python 3.13,pip install -e .[dev,tracing].Review
The pack was read four times against its sources: once in the session that built it, and three times against the full corpus in fresh Claude Code sessions that had not seen it being built. The findings from each round were fixed before the next. The last two readings found no expectation that penalises a correct answer or contradicts the sources.
One blind run
Setup:
claude-opus-5-5andclaude-haiku-5-5as targets, three repetitions each, one turn, 42 answers. The answers were judged by the default judge withclaude-sonnet-5-5, and byfact_checkfrom #112 on scenario 1. A fresh session that had not seen the pack being built read the 42 answers and fixed an expected verdict for each before any verdict was opened (label file sha2568dfb098cf2cc273486ea709c4a6314cd83e5f6f5eea4dab098b506ac2d772b6c, written 11:51:26, first verdict 11:52:07).fact_checkon scenario 1wrong, in agreement with the labels and the default judgeIn 9 of the 10 disagreements the label was pass and the judge's verdict was not. In the reader's own assessment, the judge was right or had a point in 5, the label was right in 3, and 2 are split.
No answer gave the 2026 rate. In scenario 1 all six answers gave a range between 6 000 and 8 000 kroner, and all three
claude-haiku-5-5answers said the rate is lower for a single person under 25. Rundskriv A-1/2025 has one rate for a single adult, with no age split. With 42 answers, nothing can be said about either model.Registration
The PR touches the pack module, its README,
scenarios/__init__.py, the root README table (all: 1403 to 1410), the twoall-sum assertions andCONFORMING_PACKS. #60 adds a pack and will conflict in the same registration lines; whichever merges second needs a rebase.Implemented with Claude Code (Opus 5.5); planned and reviewed with Claude.