Skip to main content

DeepPheOutputLoader

A Python toolkit to load files from directories or zip archives into a SQLite database, where the key is the filename and the value is the (optionally compressed) file content.

For the DeepPhe NLP output this stage expects — file kinds, JSON shapes, and the naming rules that later stages depend on — see DeepPhe input format.

Installation​

  1. Install the project dependencies:
uv sync

Usage​

Load Files into SQLite​

Load all files from a directory recursively:

uv run python -m dphe_db_pipeline.loader.load_to_sqlite /path/to/input/files /path/to/database

Load from a zip file:

uv run python -m dphe_db_pipeline.loader.load_to_sqlite /path/to/database --zip archive.zip

Load from a directory of zip files (recursive):

uv run python -m dphe_db_pipeline.loader.load_to_sqlite /path/to/database --zipdir /path/to/zips

Options​

  • input_dir: Directory containing files to load (optional positional, default .; ignored if --zip/--zipdir is used)
  • db_path: Path where the SQLite database will be created/opened (required positional)
  • --zip: Path to a single zip file to load
  • --zipdir: Directory tree to scan for zip files; each zip's contents are added to the database
  • --no-recursive: Do not recursively scan subdirectories (directory mode only)
  • --processes: Number of parallel worker processes when using --zipdir (default: number of CPUs)
  • --compress: Compression algorithm for content values — zstd (default), lz4, none/raw
  • --level: Compression level for the chosen algorithm (default: 1)
  • --min-compress-bytes: Only compress values at least this many bytes (default: 512)
  • --vacuum: Run VACUUM after loading to compact the database file

Examples​

Load files recursively (default):

uv run python -m dphe_db_pipeline.loader.load_to_sqlite ./data ./mydb

Load a single zip:

uv run python -m dphe_db_pipeline.loader.load_to_sqlite ./mydb --zip archive.zip

Load all zips under a directory tree:

uv run python -m dphe_db_pipeline.loader.load_to_sqlite ./mydb --zipdir /path/to/zips

How It Works​

  • The script scans the input directory (or zip archive) for files
  • For each file:
    • Key (filename): Relative path from the input directory
    • Value (content): Complete file content, optionally compressed (zstd or lz4)
    • Encoding: Tracks whether content is raw, zstd, or lz4
  • Files are stored in a SQLite database at the specified path
  • When loading from a zip directory, one worker process per CPU is used by default (override with --processes)
  • A lock prevents concurrent writes so no data is lost

Database Schema​

CREATE TABLE files (
    filename TEXT PRIMARY KEY,
    content  BLOB NOT NULL,
    encoding TEXT NOT NULL DEFAULT 'raw'
)

Writing the same key twice simply overwrites the existing row (SQLite INSERT OR REPLACE).

Stage 3 later adds its own tables to this same database file. See Output database for the complete schema.

Querying the Database​

Use dphe_db_pipeline.loader.tools.query_sqlite to verify and retrieve data.

Count total files​

uv run python -m dphe_db_pipeline.loader.tools.query_sqlite count ./mydb

List files​

# First 10 (default)
uv run python -m dphe_db_pipeline.loader.tools.query_sqlite list ./mydb

# First 50
uv run python -m dphe_db_pipeline.loader.tools.query_sqlite list ./mydb --limit 50

Retrieve a specific file​

# Display content preview
uv run python -m dphe_db_pipeline.loader.tools.query_sqlite get ./mydb "path/to/file.json"

# Extract to disk
uv run python -m dphe_db_pipeline.loader.tools.query_sqlite get ./mydb "path/to/file.json" -o output.json

Query by prefix​

uv run python -m dphe_db_pipeline.loader.tools.query_sqlite prefix ./mydb "PATIENT_ID_"

Extract a Sample Database​

To extract the first 100 patients into a smaller SQLite database:

uv run python -m dphe_db_pipeline.loader.tools.extract_100_patients \
  output/databases/individual/deepphe.sqlite3 \
  output/databases/individual/deepphe_100.sqlite3

See EXTRACT_100_PATIENTS.md for full details.

Output​

The load script provides a summary showing:

  • Total files found
  • Successfully loaded files
  • Number of errors
  • Total bytes loaded
  • Database location