Skip to content

Latest commit

 

History

History
139 lines (105 loc) · 6.29 KB

getting_started.md

File metadata and controls

139 lines (105 loc) · 6.29 KB

Onboarding Guide

This guide gets a new GlossAPI contributor from clone → first extraction with minimal detours. Use it alongside the Quickstart recipes once you're ready to explore specialised flows.

Checklist

  • Python 3.11–3.14 (requires-python = ">=3.11,<3.15")
  • Recent pip (or uv)
  • A Rust toolchain when you build from a checkout: the root wheel compiles and bundles the cleaner, noise and PDF font-repair extensions (rust-toolchain.toml pins the qualified toolchain)
  • Optional: an NVIDIA GPU with a CUDA driver, or Apple Silicon, for Docling layout and OCR

On fresh Linux hosts, make these assumptions explicit instead of relying on shell startup files:

  • prefer a stable final CPython, not a prerelease distro build
  • keep ~/.local/bin on PATH if uv was installed with pip install --user uv
  • keep ~/.cargo/bin on PATH if Rust was installed with rustup

Install GlossAPI

Recommended setup

git clone https://github.com/eellak/glossAPI.git
cd glossAPI
./dependency_setup/setup_glossapi.sh                       # core only, in .venv_glossapi_core
./dependency_setup/setup_glossapi.sh layout --device cpu   # optional: adds Docling layout (cpu, cuda126, cuda132 or mps)

The script installs into its own virtual environment but cannot activate it in your shell. Activate it before any other command on this page:

source .venv_glossapi_core/bin/activate
python -m pip --version
glossapi capabilities --json

# Optional OCR, in this same environment:
python install_glossapi.py ocr --device cpu

Add --venv PATH to choose the environment (then activate it with source PATH/bin/activate; the GLOSSAPI_VENV environment variable does the same), --editable for an editable install, --run-tests to install the test tools and run the suite, and --dry-run to print every command without running it. The OCR installer takes --device cpu (or mps / cuda126 / cuda132) and installs into the interpreter that runs it, so run it from the activated environment; see OCR setup. The full package list is in Dependency setup.

Manual editable install

For a regular source install (including all three bundled Rust extensions):

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install .                  # core: PDFium, layout-aware Markdown, clean, section, annotate, export
# Optional, in the same environment:
python -m pip install '.[ocr]'           # Greek OCR toolkit and its inference stack
python -m pip install '.[layout]'        # Docling layout and math/code enrichment
glossapi --help
glossapi capabilities --json
glossapi doctor

Install Rust (for example with rustup) and a platform C linker first: Xcode Command Line Tools on macOS, or build-essential on Debian/Ubuntu. pip install . compiles cleaner, noise and PDF repair from rust/; no separate native installer or development overlay is needed. With rustup, the checkout's rust-toolchain.toml selects the qualified toolchain; the crates declare Rust 1.85 as their minimum. Use the device-aware installer above to choose CPU/CUDA/MPS torch wheels explicitly. OCR model assets are fetched by the toolkit on first use; allow network access or prewarm the caches described in OCR setup.

To build distributable source and wheel archives, install build, then run python -m build from the checkout. The wheel bundles the native extensions for its build platform; build a separate wheel for each target platform. For development, an editable install is also available:

python -m venv .venv && source .venv/bin/activate
pip install -U pip
pip install -e .

This builds the Rust extensions needed for Corpus.clean() and the noise metrics. Re-run pip install -e . (or python install_glossapi.py core --editable) after pulling changes that touch rust/. glossapi capabilities reports a cleaner or noise extension built from older source as missing, with the detail stale native module ...; rebuild core, and glossapi doctor reports it as an error.

Adding packages later

In an environment that already has glossAPI, glossapi install --device cpu --package download prints the exact install commands for the named packages without running them; add --yes to run them. Each command's output is shown on stderr while it runs.

GPU prerequisites (optional)

setup_glossapi.sh layout --device cuda126 (or cuda132, mps) installs the matching torch build before Docling. The device name selects the wheel source only; it does not choose the device of a later run. To check that torch sees the GPU (in the activated environment):

python -c "import torch; print(torch.cuda.is_available())"

First run (lightweight corpus)

Run this from the checkout, in the activated environment. It saves a small script and runs it; keep the if __name__ == "__main__": guard, because extraction workers start with spawn and import the script again (code piped to python - cannot be imported again, so its workers fail):

cat > lightweight_run.py <<'PY'
from pathlib import Path
from glossapi import Corpus


def main():
    input_dir = Path("samples/lightweight_pdf_corpus/pdfs")
    output_dir = Path("artifacts/lightweight_pdf_run")
    output_dir.mkdir(parents=True, exist_ok=True)
    corpus = Corpus(input_dir, output_dir)
    corpus.extract(input_format="pdf")  # safe PDFium backend, layout_v2 Markdown


if __name__ == "__main__":  # extraction workers start with spawn and import this file again
    main()
PY
python lightweight_run.py
  • Inspect artifacts/lightweight_pdf_run/markdown/ and compare with samples/lightweight_pdf_corpus/expected_outputs.json.
  • Run pytest tests/test_corpus_phases_lightweight.py for a reproducible regression check on the same two PDFs (extract, clean and section; no GPU or Docling).

Next steps