[DevOps]: Add Chrysalis ingestion wrapper and deploy - #169
Conversation
|
@tomvothecoder Once I am clear on the boundaries to the term "NERSC ingestion wrapper", I should be able to comprehent "Chrysalis ingestion wrapper". The term "scheduler-agnostic" refers to Jenkins? (I always considered cron to be universal...). |
TonyB9000
left a comment
TonyB9000
left a comment
There was a problem hiding this comment.
Configures a call to the hpc_archive_ingestor. Understandable.
What process sets "SIMBOARD_API_BASE_URL" and "SIMBOARD_API_TOKEN"?
|
Hmmmm. The "Tom Requested your review" took me to the page with 7 files to examine, each with a "submit-review" option. As soon as I completed the first one, all 7 vanished... |
Accidentally tagged you for review. I meant to assign this PR you. It is fixed now. |
|
@tomvothecoder "Accidentally tagged you for review". OK, (I think colleges should offer a master's program in github). |
|
Chrysalis and other non-NERSC sites require upload-based ingestion rather than path-based ingestion, so follow-up work is tracked in #207 for a state-first HPC upload flow with DB-backed dedupe parity. |
|
Using the "upload-based' vs "path-based" terminology, my thought was that when the NERSC upload-receiving system was deliverd an upload from a non-NERSC system, it could open it in the existing NERSC PA-directory under (say) "From_crysalis/<new_exec_ids>" and then process it with the existing "path-based" codes - assuming PACE would not interfere with it (and vice-versa). But on second thought, to avoid PACE crossing, it would be best to open it in a separate "PACE-unaware" directory. |
|
@tomvothecoder I am preparing to exercise "hpc_upload_archive_ingestor.py" on chrysalis, to see the logs and flow (in dry-run) in action, discover parameter faults, etc. QUESTION: Although, on NERSC, the backend ingestion is "path-based" (returnsp paths for ingestion), it could in principle run the "https-transfer-based" codes just as easily. I might try a dryrun on NESRC/Perlmutter first, since that configuration is already a known item. Then, differences in behavior on chrysalis would stand out. Does that make sense? |
22a1a88 to
feae197
Compare
|
@tomvothecoder Apologies if I'm doing this wrong. I attempted to test the "hpc_upload" on NESRC, thinking "--help" might be helpful. To get started, I needed an environment where I could install things, so: |
|
Hey Tony, happy to help and no apologies needed.
SimBoard defines the Python backend dependencies in You can run
I'd checkout this branch now that I've rebased it on the latest The |
|
@tomvothecoder I get the latest stuff - but I have made progress. My latest run_script (NERSC dry_run test) says: The output indicates that I am missing "archive_root” and “has_api_token”. By examining the "nersc" "_build_config" function, I can see what variables exist to push into the environment. I'll checkout branch #169 on both NERSC and Chrysalis to do comparisons in outputs. |
|
@tomvothecoder git gets me again: You wrote: "I'd checkout this branch now that I've rebased it on the latest main commit." is "this branch" off of main, as you had advised? Or is it off of a fork?? ((test_simboard) ) (base) [ac.bartoletti1@chrlogin1 simboard]$ git branch -a
When I get too confused, I do a clean "git clone". Then I can do one of these: Which is appropriate in this case? |
|
@tomvothecoder If I pull down a branch off of someone's fork, am I in that fork, or can I pull that into a new branch of my local main? The persistence of branches and forks, between local and remote, is a bit of a mystery. |
This branch ( Something like this (I did not verify correctness): git remote add tomvothecoder https://github.com/tomvothecoder/simboard
git checkout tomvothecoder feature/154-ingestion-ites I usually work directly on upstream and not fork when possible, but in this case I use a fork for separate testing purposes. |
|
@tomvothecoder Sorry, I guess I must pull from your fork. Quick test: I cd to ``/home/ac.bartoletti1/gitrepo/simboard/backend" and issue
The result: |
|
@tomvothecoder The line in "chrysalis.sh" script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"; echo $script_dir clearly wont work for defining "backend_root" as backend_root="$(cd "${script_dir}/../../../.." && pwd)" I will modify chrysalis.sh to provide a "backend_root" that does not depend upon the user location., at least for test purposes. |
|
@tomvothecoder Works better now that it can find "backentd/apps" When I use this for "chrysalis.sh": and issue these exports: I get: I guess, even "dry_run" requires real URLs and API_tokens. That is because we need "state" up front. |
|
Hi @tomvothecoder The document also says: One-case-per-request rule:
The term "alongside the archive" is a bit ambiguous. Would this be accurate?
Or am I misunderstanding the intent? |
Great to see the progress! Yes, the dry run needs to query the SimBoard database via the REST API. I will send the
Your info sounds more accurate, thanks for the suggestion. Can you point me to the source document with this info? I will update it. |
|
@tomvothecoder Running "chrysalis.sh" with the full (DRY_RUN) parameters yieded the following summary (folded for readability): Questions that arise:
Observation: The bulk of work getting to this point involved stuffing the right ENV VARS and having created an environment where misc modules like "dateutils" could be installed. On Chrysalis, I performed On NERSC/Perlmutter, I simply replaced "python3.12" with "python3.11". I intend to perform the same test on Perlmutter, just to exercise the mechanisms of networking. |
|
@tomvothecoder For comparison, running the equivalent commands on perlmutter (swapping our parameters where necessary), we obtain the summary: I suppose I should re-run the chrysalis test, using "OLD_PERF" as the root_PA directory. It is HUGE. |
Since this is a first-time dry-run on the Chrysalis
That exact counter is not in So yes: a test expecting that exact field is probably unrealistic or stale.
The ingestor script only asks SimBoard for enough existing ingestion state to decide which archive cases and their executions are new and may be candidates for ingestion. It does not fetch, return, or summarize the full database state. It also does not show the database query because the query is behind the SimBoard API, not inside the ingestor script. So this is not a Chrysalis-specific DB query in the ingestor. It is an API request filtered by the configured If more detail is needed, the API response or ingestor summary would need to be expanded to include counts like total known cases, known execution IDs, skipped known cases, and machine filter used. Happy for you to open a new GitHub issue to expand logging in https://github.com/E3SM-Project/simboard/blob/main/backend/app/scripts/ingestion/nersc_archive_ingestor.py and https://github.com/E3SM-Project/simboard/blob/main/backend/app/scripts/ingestion/hpc_upload_archive_ingestor.py.
They are dry-run logging counters.
The point is to avoid massive logs when many candidate cases are found. It does not change which cases are candidates or which cases would be ingested. |
I don't think this is going to work yet as the directory structure of "OLD_PERF" is different from "performance_archive". We might also want to be targeted in what we ingest from "OLD_PERF". This will require guidance Rob/Jill. |
|
@tomvothecoder Job Completed after ~20 minutes. 15 new successes, which I assume include the 11 past failures (not trivial to tell without a more structured summary). The last lines of the log are as follows: |
I merged #308 and updated the upload limit on the NERSC backend to fix this. Nothing needed on your end.
You can try cross-checking the failing case ids against the latest logs here to see if they are not listed as successful. You have the failing case ids listed here: #169 (comment). I will also send you read-only SQL credentials and instructions over email to try SQL queries for this information. I recommend using an agentic AI code tool like GitHub Copilot (E3SM provides access with a seat on the org account) because they have the context of the repository to develop SQL queries for you to run. My SQL skills are rusty so that is what I'm doing. Of course, it's another software you'd need to learn. |
|
@tomvothecoder It appears that all 11 are successful - at least no errors were generated. I isolated each case_id and fed them to a script that does: which is an admittedly crude way to review the logs. The results are: |
|
@tomvothecoder Progress Report: After running for about 30 minutes, the directory collection/assessment accumulated just over 2000 new cases, and consequently accepted over 5700 executions (2025-01 through 2026-09). It has spent most of the past 2 hours packaging and uploading the new archives, having completed 1138 of the 2000. It should complete by 4 PM (PST, or PDT, or whatever time it is really...) |
|
@tomvothecoder The chrysalis archive run completed, "duration_seconds=21458.032" (6 hrs). The complete logfile (39989 lines) is here: The last 4 (run-summary) lines are here: (easier to read): |
|
ASIDE: A breakdown of the log file: |
Can get the names of the 6 cases that failed and their error codes? I wonder if the upload limit needs to be further increased. |
|
I'll examine the log-file for details. They all appear to be "Bad Gateway" or "Gateway-Timeout" issues: Additionally, out of these "archive_run" daily log-files, only the very first one had these errors: Of course, the first one took a bit longer than the others: (6 hours, versus a few minutes). |
|
Here are the summary reports generated daily from the output logs: |
|
Contents of latest "Staging Gains" file: |
@TonyB9000 I confirmed all five distinct cases were ingested successfully. The May case has two failed retry attempts, which is why the original log contained six failure entries. Database ResultsWITH wanted(case_path) AS (
VALUES
('/lcrc/group/e3sm/PERF_Chrysalis/OLD_PERF/2025-01/performance_archive_chrysalis_e3sm_2025_01_07_00_45_55/ac.kai.zhang/v3.LR.piClim-histaer'),
('/lcrc/group/e3sm/PERF_Chrysalis/OLD_PERF/2025-03/performance_archive_chrysalis_e3sm_2025_03_06_00_46_47/ac.ambradl/masscons.LR.nlmap1'),
('/lcrc/group/e3sm/PERF_Chrysalis/OLD_PERF/2025-05/performance_archive_chrysalis_e3sm_2025_05_17_00_46_48/ac.szhang/20250502.v3.LR.IGPRCPROG.TEST2.DATM.CPLHIST.chrysalis'),
('/lcrc/group/e3sm/PERF_Chrysalis/OLD_PERF/2025-06/performance_archive_chrysalis_e3sm_2025_06_12_00_47_54/ac.ambradl/v3.LR.F2010.ei'),
('/lcrc/group/e3sm/PERF_Chrysalis/OLD_PERF/2025-08/performance_archive_chrysalis_e3sm_2025_08_21_00_48_54/ac.zguo/sens1022_77')
),
ingestion_summary AS (
SELECT
i.source_reference AS case_path,
count(*) AS ingestion_records,
max(i.created_at) AS last_ingested_at,
array_agg(DISTINCT i.status::text) AS statuses,
sum(i.created_count) AS executions_created,
sum(i.duplicate_count) AS executions_duplicate,
sum(i.error_count) AS ingestion_errors
FROM ingestions i
JOIN machines m ON m.id = i.machine_id
JOIN wanted w ON w.case_path = i.source_reference
WHERE m.name = 'chrysalis'
AND i.source_type = 'hpc_upload'
GROUP BY i.source_reference
)
SELECT
w.case_path,
COALESCE(s.ingestion_records, 0) AS ingestion_records,
s.last_ingested_at,
s.statuses,
COALESCE(s.executions_created, 0) AS executions_created,
COALESCE(s.executions_duplicate, 0) AS executions_duplicate,
COALESCE(s.ingestion_errors, 0) AS ingestion_errors,
CASE WHEN s.case_path IS NULL THEN 'not_ingested' ELSE 'ingested' END AS result
FROM wanted w
LEFT JOIN ingestion_summary s USING (case_path)
ORDER BY w.case_path;
Result: 5/5 cases ingested successfully; 8 executions created; no ingestion errors recorded. Checking Upload SizeI also checked the upload sizes from Retries reuse the same prebuilt archive, so each case is listed once rather than summing retry sizes. All uploads were well below the configured 325 MiB NGINX request-body limit; upload size was therefore not the cause of the 502/504 responses. I think we can label these failures as transient network issues and deal with it later if it re-appears frequently. Subsequent retries should pick up these failures, as shown below (default is grep 'event=case_upload_attempt' SBCS-archive-chrysalis-2026090* \
| grep -E \
'v3\.LR\.piClim-histaer|masscons\.LR\.nlmap1|20250502\.v3\.LR\.IGPRCPROG\.TEST2\.DATM\.CPLHIST\.chrysalis|v3\.LR\.F2010\.ei|sens1022_77' \
| perl -ne 'print "$1\tattempt=$2\tupload_bytes=$3\n" if /case_path=([^ ]+).*attempt=(\d+).*archive_bytes=(\d+)/'
|
|
@TonyB9000 This PR is in good shape. I think any additional logging improvements should be done in a new PR to prevent the scope of this PR from getting too large. We can also address the production cron jobs later in #304. I will do a final cleanup and review. Let me know if there's anything else that needs to be updated first. |
tomvothecoder
left a comment
tomvothecoder
left a comment
There was a problem hiding this comment.
My initial self-review, then I will assign to agent to review.
|
@tomvothecoder I made some changes to log naming/location in the launch script. I don;t think that will be a problem. It is working correctly now, so I will push that soon. |
Thanks. I just push a new commit: |
There was a problem hiding this comment.
🟡 Changes recommended
The scheduling and logging design has operational risks, and several documentation sections are incorrect or corrupted.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds config-driven HPC ingestion for Chrysalis while preserving NERSC path-based ingestion.
Changes:
- Adds shared site launcher and Chrysalis/NERSC configurations.
- Adds remote-state-aware dry runs and tests.
- Expands ingestion, deployment, and database-access documentation.
File summaries
| File | Description |
|---|---|
backend/app/scripts/README.md |
Documents operational ingestion workflows. |
backend/app/scripts/ingestion/archive_ingestor_core.py |
Adds remote-state dry-run configuration. |
backend/app/scripts/ingestion/archive_workflow.py |
Adjusts dry-run authentication validation. |
backend/app/scripts/ingestion/hpc_upload_archive_ingestor.py |
Supports offline and remote-state dry runs. |
backend/app/scripts/ingestion/nersc_archive_ingestor.py |
Supports offline and remote-state dry runs. |
backend/app/scripts/ingestion/sites/chrysalis.config |
Defines Chrysalis ingestion settings. |
backend/app/scripts/ingestion/sites/crontab.example |
Adds generic collection schedules. |
backend/app/scripts/ingestion/sites/nersc.config |
Defines NERSC ingestion settings. |
backend/app/scripts/ingestion/sites/nersc.crontab.example |
Removes superseded NERSC schedule. |
backend/app/scripts/ingestion/sites/nersc.sh |
Removes superseded NERSC wrapper. |
backend/app/scripts/ingestion/sites/site_ingestion_launcher.sh |
Adds the shared site launcher. |
backend/tests/features/ingestion/test_archive_workflow.py |
Tests dry-run preconditions and logging. |
backend/tests/features/ingestion/test_hpc_upload_archive_ingestor.py |
Tests upload-ingestor dry-run behavior. |
backend/tests/features/ingestion/test_nersc_archive_ingestor.py |
Tests NERSC offline dry runs. |
backend/tests/features/ingestion/test_site_collection_launcher.py |
Tests launcher configuration behavior. |
docs/README.md |
Links database-access documentation. |
docs/architecture/metadata-ingestion.md |
Documents dry-run and scan configuration. |
docs/deploy/README.md |
Adds the database-access guide. |
docs/deploy/nersc-spin-runbook.md |
References the replacement launcher. |
docs/deploy/read-only-database-access.md |
Documents read-only PostgreSQL access. |
docs/github-issues/154-ingestions/codex-implementation-plan.md |
Records the Chrysalis rollout plan. |
docs/github-issues/154-ingestions/ingestion-sites-onboarding.md |
Adds site onboarding guidance. |
docs/github-issues/154-ingestions/simboard-hpc-ingestion-architecture.png |
Adds an ingestion architecture diagram. |
mkdocs.yml |
Adds the database guide to navigation. |
Review details
Suppressed comments (9)
backend/app/scripts/README.md:37
- The file tree omits the existing
lcrc_v3_hpss_linker.py, even though the same README lists it as a primary entry point below. Keep this inventory consistent with the directory it documents.
│ ├── lcrc_v3.sh
│ └── lcrc_v3_archive_ingestor.py
backend/app/scripts/README.md:242
- This heading labels the following section as the NERSC archive ingestor, but the content and command describe
diagnostics_link_scanner.py. This makes the diagnostics operational instructions appear to belong to the wrong runner.
## NERSC Archive Ingestor
backend/app/scripts/README.md:268
- These archive-ingestor settings do not apply to the diagnostics scanner described by this section; that scanner reads only
MACHINE_NAME,DRY_RUN, and, for live runs, the API URL/token. The archive variable list and notes have displaced the scanner's LCRC invocation and operational behavior, so following this section will not configure the documented command correctly.
- `SCAN_MODE` (`staging` or `archive`, default `staging`)
- `PERF_ARCHIVE_ROOT` (default `/performance_archive` for `SCAN_MODE=staging`)
- `OLD_PERF_ARCHIVE_ROOT` (default `/OLD_PERF` for `SCAN_MODE=archive`)
- `MACHINE_NAME` (default `perlmutter`)
- `DRY_RUN` (default `true`)
docs/github-issues/154-ingestions/ingestion-sites-onboarding.md:56
- This link points to a file that does not exist; the token guide is under
docs/deploy/hpc-api-token-authentication.md.
See `docs/hpc_api_token_authentication.md` for service account and API token setup.
docs/github-issues/154-ingestions/ingestion-sites-onboarding.md:127
- This reference repeats the nonexistent underscore-style path. Link to the deployed token guide instead.
- API token docs: `docs/hpc_api_token_authentication.md`
docs/github-issues/154-ingestions/ingestion-sites-onboarding.md:84
- The new runners do not use a writable local state file; deduplication state and archive checkpoints are fetched from and persisted through the SimBoard API. This checklist item, and the
state-file behaviorwording at line 96, direct operators to validate a component that does not exist.
- The state file location is writable and persists across runs.
docs/github-issues/154-ingestions/codex-implementation-plan.md:52
- The implementation has no
STATE_PATH; state is database-backed and accessed through the ingestion-state and archive-checkpoint APIs. The same stale state-file model recurs at lines 73, 83, and 168, so the plan should describe remote state instead of asking site wrappers to configure a nonexistent path.
- set `STATE_PATH`
docs/github-issues/154-ingestions/codex-implementation-plan.md:152
- The upload runner introduced here calls
/ingestions/from-hpc-upload, not the browser-oriented/ingestions/from-uploadendpoint. Listing the latter as sufficient gives the wrong API contract for the planned Chrysalis flow.
- Existing SimBoard `/ingestions/from-path` and `/ingestions/from-upload` APIs
are sufficient for initial rollout planning.
backend/app/scripts/README.md:412
- The sentence promises HPSS linker apply and verification commands, but the following content is an unrelated archive-ingestor environment list. Restore the linker commands so operators can actually perform the documented apply and idempotency check.
After review, apply the links and rerun the dry run to confirm idempotency:
- `SIMBOARD_API_BASE_URL`
- Files reviewed: 23/24 changed files
- Comments generated: 8
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| # Archive scan: run daily at 12:00 UTC. The site config supplies the default | ||
| # lower bound. Override `ARCHIVE_YEAR_START` or `ARCHIVE_YEAR_END` in the cron | ||
| # environment for a differently scoped scan. Values may use YYYY or YYYY-MM. | ||
| 0 12 * * * cd "${SIMBOARD_ROOT}/operations" && "${SIMBOARD_ROOT}/repository/simboard/backend/app/scripts/ingestion/sites/site_ingestion_launcher.sh" chrysalis archive |
There was a problem hiding this comment.
@tomvothecoder It occurs to me that we need a separate lock for the archive_run. It should not be paying any attention to the staging runs - but be fully independent. It did not occur to me that they may contend for network, etc (never tested that).
| ts="$(date -u +%Y%m%d_%H%M%S)" | ||
| LOG_FILE="${SIMBOARD_WORKDIR}/SBCS-${scan_mode}-${site}-${ts}.log" | ||
| printf '[%s] launcher started: site=%s scan_mode=%s dry_run=%s\n' \ | ||
| "$(date -Is)" "${site}" "${scan_mode}" "${dry_run_normalized}" >> "${LOG_FILE}" |
| @@ -0,0 +1,20 @@ | |||
| # Example crontab entries for a generic site collection launcher. | |||
| # Install with something like: | |||
| # crontab backend/app/scripts/ingestion/sites/<site>.crontab.example | |||
|
@TonyB9000 Follow-up work is tracked in focused issues so this PR can merge as-is. I will pull the latest
These are independent operational refinements; the Chrysalis ingestion functionality is in a good state for this PR to merge. |
Description
This PR replaces the one-off NERSC host wrapper with a config-driven site collection launcher and adds Chrysalis support for remote, upload-based metadata ingestion. The launcher keeps scheduler/site setup in shell configuration while selecting the appropriate shared Python runner: path-based ingestion at NERSC and upload-based ingestion at Chrysalis.
sites/site_ingestion_launcher.sh, which loads a named site config, applies staging/archive scan mode, serializes runs with a lock, loads protected API configuration when required, and invokes the configured Python module.sites/chrysalis.configfor the Chrysalis archive roots,chrysalismachine identity, upload ingestor, API/token-file locations, and archive scan lower bound.sites/nersc.configand removes the supersededsites/nersc.shand NERSC-specific crontab example.hpc_upload_archive_ingestor.pyfor non-NERSC sites, packaging each eligible case and submitting it to/api/v1/ingestions/from-hpc-upload; NERSC continues to usenersc_archive_ingestor.py.DRY_RUN_USE_REMOTE_STATE=falsesupports credential-free offline discovery.Task
Finish turning this branch into a deployable ingestion path for Chrysalis first, then use the same pattern for other sites once access is available.
SIMBOARD_API_BASE_URLandSIMBOARD_API_TOKENshould be stored and injected in the Chrysalis Jenkins job.Checklist
Deployment Notes (if any)
No special deployment steps.