Skip to content

scripts.md

Latest commit

 

History

History
167 lines (117 loc) · 4.85 KB

scripts.md

File metadata and controls

167 lines (117 loc) · 4.85 KB
Raw

Backend Scripts

Backend utility scripts live in backend/scripts/. This public page documents safe local-development uses and points restricted operational scripts to the private operations documentation.

Prefer Makefile targets for repeated workflows so environment loading and Docker wrapping stay consistent. See ../make_tasks.md.

Safe Local Scripts

IIIF thumbnail generation

Thumbnail generation preserves the provider and image identifier in IIIF Image API references. Presentation manifests are resolved in the background, including CONTENTdm manifests: a compound object can contain page images with different identifiers. When a manifest has no explicit thumbnail, its first canvas's image service supplies a bounded rendition. This also supports services whose paths do not contain iiif, such as Loris, where the image resource ID may be a catalog page rather than an image URL.

After changing thumbnail resolution, regenerate selected records in the local development database with:

make prime-thumbnail-cache RESOURCE_IDS="example-record-id another-record-id" PRIME_FORCE=1

This requires the local database and cache services. Force regeneration retries records with existing thumbnail state or cached images. See ../make_tasks.md for local cache workflows. Deployed backfills belong in the restricted operations documentation.

process_allmaps.py

Processes and generates Allmaps annotations for resources in the local database.

python scripts/process_allmaps.py --item-id "example-id"
python scripts/process_allmaps.py --all

populate_relationships.py

Populates relationship data between resources.

python scripts/populate_relationships.py
make populate-relationships

See relationships.md.

generate_fast_embeddings.py

Generates and stores embeddings for FAST gazetteer data. Requires a configured OpenAI API key in the local environment.

python scripts/generate_fast_embeddings.py

run_migration.py

Runs script-based local migrations.

python scripts/run_migration.py add_fast_gazetteer

import_fast.py

Imports OCLC FAST geographic records into the local database.

python scripts/import_fast.py

clear_cache.py

Clears local Redis cache data for development and testing.

python scripts/clear_cache.py

clear_cache_by_type.py

Clears local cache entries by tag type.

python scripts/clear_cache_by_type.py search
python scripts/clear_cache_by_type.py all

test_gazetteer_api.py

Exercises gazetteer API endpoints against a configured base URL.

python scripts/test_gazetteer_api.py --base-url http://localhost:8000/api/v1

manage_analytics_storage.py

Maintains local analytics partitions, rollups, and retention tables.

cd backend
python scripts/manage_analytics_storage.py --mode maintenance
python scripts/manage_analytics_storage.py --mode size-report
python scripts/manage_analytics_storage.py --mode ensure

From the project root:

make analytics-maintenance
make analytics-size-report

Restricted Operational Scripts

Some scripts are intended for deployment bootstrap, remote maintenance, backup, restore, deployed cache handling, or incident response. Public docs must not include hostnames, destination names, command blocks, secret names, storage layouts, credential setup, or recovery procedures for those scripts.

Use the restricted operations documentation for:

  • deployment bootstrap scripts;
  • remote backup and restore scripts;
  • deployed search/cache/database maintenance;
  • deployed analytics maintenance;
  • deployed service-tier and API-key operations;
  • incident response and recovery playbooks.

API Rate Limiting And Service Tiers

Rate limiting for the public API is enforced by middleware backed by Redis. Tables and seed data are created by the standard migration script:

.venv/bin/python scripts/run_migrations.py

Public docs may describe the implementation model and local testing. Production key provisioning, deployed tier changes, monitoring, and troubleshooting are restricted operations material.

For code-level context, see:

Maintenance Checklist

  • Add a Makefile target when a script becomes part of a repeated local development workflow.
  • Document new public-safe target variables in ../make_tasks.md.
  • Add or update backend tests when a script mutates database, cache, index, or external service state.
  • Keep destructive scripts guarded by clear names, dry-run flags, or Makefile comments.
  • Do not commit generated outputs from tmp/, logs/, data volumes, cache directories, or Python/Node build artifacts.