ffx/requirements/tests.md

# Test Rewrite

This file captures the structure executed by `tests/legacy_runner.py` today and
defines the target shape for a complete rewrite.

Detailed product rules for source-to-target subtrack mapping live in
`requirements/subtrack_mapping.md`. This file describes only how tests cover
that area.

## Interpreter Requirement

- Agents shall run Python-side test commands with `~/.local/share/ffx.venv/bin/python`.
- This applies to the legacy harness, `unittest`, `pytest`, helper scripts, and `python -m ffx ...` test invocations.
- Agents shall not silently substitute `python`, `python3`, or another interpreter for Python-side test work.
- If `~/.local/share/ffx.venv/bin/python` is missing or not executable, agents shall stop and report the missing venv instead of continuing with Python-side test execution.

## Current Harness

- Entrypoint: `~/.local/share/ffx.venv/bin/python tests/legacy_runner.py run`
- Runner style: custom Click CLI, not `pytest` or `unittest`
- Commands:
  - `run`: discover scenario files, instantiate each scenario, run yielded jobs
  - `dupe`: helper command that creates duplicate media fixtures; not part of the test run
- Filters: `--scenario`, `--variant`, `--limit`
- Shared context:
  - builds one mutable dict for the whole run
  - installs loggers and writes `ffx_test_report.log`
  - creates `ConfigurationController` eagerly
  - tracks only passed and failed counters
- Discovery:
  - scenario files: `tests/legacy/scenario_*.py`
  - combinators: `glob + importlib + inspect` by filename convention
  - ordering: implicit glob order, no explicit sorting
- Skip behavior:
  - Scenario 4 is skipped when `TMDB_API_KEY` is missing
  - only `TMDB_API_KEY_NOT_PRESENT_EXCEPTION` is caught at scenario construction time

## Current Scenarios

- `1`: `tests/legacy/scenario_1.py`
  - focus: basename generation without pattern lookup or TMDB
  - inputs per job: `1`
  - jobs: `140`
  - expected failures: `0`
  - execution: build one synthetic source file, run `~/.local/share/ffx.venv/bin/python -m ffx convert`, assert filename selectors only
  - selectors executed: `B`, `L`, `I`
  - selectors defined but not executed: `S`, `R`
- `2`: `tests/legacy/scenario_2.py`
  - focus: conversion matrix over media layouts, dispositions, tags, and permutations
  - inputs per job: `1`
  - jobs: `8193`
  - expected failures: `3267`
  - execution: build one synthetic source file, run `~/.local/share/ffx.venv/bin/python -m ffx convert`, probe result with `FileProperties`, assert track layout and selected audio and subtitle metadata
  - selectors executed: `M`, `AD`, `AT`, `SD`, `ST`
  - selectors defined but not executed: `MT`, `AP`, `SP`, `J`
- `4`: `tests/legacy/scenario_4.py`
  - focus: pattern-driven batch conversion with SQLite state and live TMDB naming
  - inputs per job: `6`
  - jobs: `768`
  - expected failures: `336`
  - execution: build six synthetic preset files, recreate temp SQLite DB, insert show and pattern, run one batch convert command via `~/.local/share/ffx.venv/bin/python`, query TMDB during assertions
  - selectors executed: `M`, `AD`, `AT`, `SD`, `ST`
  - selectors defined but not executed: `MT`, `AP`, `SP`, `J`
  - notes:
    - uses `MediaCombinator6` only
    - issues live HTTP requests through `TmdbController` with no request cache

## Current Combinator Families

- scenario files discovered: `3`
- basename combinators discovered: `2`
- media combinators discovered: `8`
- media tag combinators discovered: `3`
- disposition combinator 2 variants: `4`
- disposition combinator 3 variants: `5`
- track tag combinator 2 variants: `4`
- track tag combinator 3 variants: `5`
- indicator variants: `7`
- label variants: `2`
- show variants: `3`
- release variants: `3`
- permutation 2 variants: `2`
- permutation 3 variants: `3`

## Current Totals

- full run without TMDB: `8333`
- full run with TMDB: `9101`
- Scenario 4 generated source files: `4608`
- Scenario 4 live TMDB episode queries: `4608`

## Current Behavior Areas

- output basename rules for label, season and episode indicator, show name, and release suffix combinations
- track layout normalization across the eight media combinator shapes from `VA` through `VAASSS`
- two-track and three-track disposition edge cases, including intentional failure cases
- two-track and three-track track-tag preservation checks, including checks that sort results by source identity
- container-level media tag handling
- pattern-backed conversion against a temporary SQLite database
- TMDB-assisted episode naming for batch conversion

## Structural Findings

- The suite is process-heavy: most jobs run `ffmpeg` to generate a fixture and then spawn the FFX CLI as a subprocess.
- The suite is integration-first and has almost no isolated unit-level coverage for pure logic.
- The base `Combinator` class is a placeholder and is not the real abstraction boundary used by the suite.
- Many combinator methods are placeholders: there are `25` `pass` statements across the current test modules.
- Several assertion families are never executed because scenario selector dispatch is incomplete.
- Scenario comments mention a Scenario 3, but no `scenario_3.py` exists.
- `tests/legacy/_basename_combinator_1.py` is effectively orphaned because discovery only matches `basename_combinator_*.py`.
- `tests/legacy/disposition_combinator_2_3 .py` contains an embedded space in the filename and is still part of discovery.
- Expected failures are validated only as subprocess return-code matches, not as specific error types or messages.
- The current suite depends on `ffmpeg`, `ffprobe`, SQLite, the local Python environment, and for Scenario 4 a live TMDB API key plus network access.

## Rewrite Target

- Replace the custom Click harness with a standard test runner, preferably `pytest`.
- Split the suite into explicit layers: unit, integration, and optional external-system tests.
- Keep unit tests as the default path and make them runnable without `ffmpeg`, `ffprobe`, TMDB, or a user config directory.
- Model discovery explicitly in code instead of relying on glob-plus-reflection naming conventions.
- Convert the current Cartesian-product combinators into readable parametrized cases grouped by behavior area.
- Preserve the current behavior areas, but represent them with targeted cases instead of thousands of opaque variant IDs.
- Make every assertion family explicit and executable; there must be no selector that is produced but never consumed.
- Replace live TMDB access with fixtures or mocks in normal runs; any live-contract test must be opt-in.
- Replace ad hoc subprocess return-code checks with assertions on typed exceptions, stderr content, or structured outputs.
- Provide small reusable media fixtures or fixture builders so only a narrow integration slice needs `ffmpeg`-generated media.
- Make database tests self-contained and fast through temporary databases and direct controller-level assertions.
- Make ordering, naming, and selection deterministic so a contributor can predict exactly what will run.
- Expose a small smoke suite for quick local runs and CI, plus a separately marked slower integration suite.
- Prefer domain-oriented test modules over combinator-family modules: basename, pattern matching, metadata rewrite, track ordering, TMDB naming, CLI smoke, and failure handling.

## Rewrite Acceptance

- A default local test run finishes quickly and without network access.
- A contributor can identify which behavior a failing test covers without decoding variant strings like `VAASSS-A:D10-S:T001`.
- All current intended failure behaviors remain covered, but each one is asserted directly and readably.
- The rewritten suite can be adopted by CI without requiring live TMDB credentials.