Test Rewrite

This file captures the structure executed by tests/legacy_runner.py today and defines the target shape for a complete rewrite.

Detailed product rules for source-to-target subtrack mapping live in requirements/subtrack_mapping.md. This file describes only how tests cover that area.

Interpreter Requirement

Agents shall run Python-side test commands with ~/.local/share/ffx.venv/bin/python.
This applies to the legacy harness, unittest, pytest, helper scripts, and python -m ffx ... test invocations.
Agents shall not silently substitute python, python3, or another interpreter for Python-side test work.
If ~/.local/share/ffx.venv/bin/python is missing or not executable, agents shall stop and report the missing venv instead of continuing with Python-side test execution.

Shell Environment Requirement

Agents shall source ~/.bashrc from an interactive Bash shell before running TMDB-dependent test commands or TMDB-dependent python -m ffx ... test invocations.
Agents shall not source ~/.bashrc.d/interactive/77_tmdb.sh directly for normal test work; ~/.bashrc is the required entry point.
In automation this means agents shall use an interactive Bash invocation such as bash -ic 'source ~/.bashrc && ...', because a non-interactive bash -lc returns from ~/.bashrc before the interactive fragments are loaded.
If sourcing ~/.bashrc still does not provide required shell environment such as TMDB_API_KEY, agents shall stop and report the missing environment instead of continuing with TMDB-dependent test execution.

Current Harness

Entrypoint: ~/.local/share/ffx.venv/bin/python tests/legacy_runner.py run
Runner style: custom Click CLI, not pytest or unittest
Commands:
- run: discover scenario files, instantiate each scenario, run yielded jobs
- dupe: helper command that creates duplicate media fixtures; not part of the test run
Filters: --scenario, --variant, --limit
Shared context:
- builds one mutable dict for the whole run
- installs loggers and writes ffx_test_report.log
- creates ConfigurationController eagerly
- tracks only passed and failed counters
Discovery:
- scenario files: tests/legacy/scenario_*.py
- combinators: glob + importlib + inspect by filename convention
- ordering: implicit glob order, no explicit sorting
Skip behavior:
- Scenario 4 is skipped when TMDB_API_KEY is missing
- only TMDB_API_KEY_NOT_PRESENT_EXCEPTION is caught at scenario construction time

Current Scenarios

1: tests/legacy/scenario_1.py
- focus: basename generation without pattern lookup or TMDB
- inputs per job: 1
- jobs: 140
- expected failures: 0
- execution: build one synthetic source file, run ~/.local/share/ffx.venv/bin/python -m ffx convert, assert filename selectors only
- selectors executed: B, L, I
- selectors defined but not executed: S, R
2: tests/legacy/scenario_2.py
- focus: conversion matrix over media layouts, dispositions, tags, and permutations
- inputs per job: 1
- jobs: 8193
- expected failures: 3267
- execution: build one synthetic source file, run ~/.local/share/ffx.venv/bin/python -m ffx convert, probe result with FileProperties, assert track layout and selected audio and subtitle metadata
- selectors executed: M, AD, AT, SD, ST
- selectors defined but not executed: MT, AP, SP, J
4: tests/legacy/scenario_4.py
- focus: pattern-driven batch conversion with SQLite state and live TMDB naming
- inputs per job: 6
- jobs: 768
- expected failures: 336
- execution: build six synthetic preset files, recreate temp SQLite DB, insert show and pattern, run one batch convert command via ~/.local/share/ffx.venv/bin/python, query TMDB during assertions
- selectors executed: M, AD, AT, SD, ST
- selectors defined but not executed: MT, AP, SP, J
- notes:
  - uses MediaCombinator6 only
  - issues live HTTP requests through TmdbController with no request cache

Current Combinator Families

scenario files discovered: 3
basename combinators discovered: 2
media combinators discovered: 8
media tag combinators discovered: 3
disposition combinator 2 variants: 4
disposition combinator 3 variants: 5
track tag combinator 2 variants: 4
track tag combinator 3 variants: 5
indicator variants: 7
label variants: 2
show variants: 3
release variants: 3
permutation 2 variants: 2
permutation 3 variants: 3

Current Totals

full run without TMDB: 8333
full run with TMDB: 9101
Scenario 4 generated source files: 4608
Scenario 4 live TMDB episode queries: 4608

Current Behavior Areas

output basename rules for label, season and episode indicator, show name, and release suffix combinations
track layout normalization across the eight media combinator shapes from VA through VAASSS
two-track and three-track disposition edge cases, including intentional failure cases
two-track and three-track track-tag preservation checks, including checks that sort results by source identity
container-level media tag handling
pattern-backed conversion against a temporary SQLite database
TMDB-assisted episode naming for batch conversion

Structural Findings

The suite is process-heavy: most jobs run ffmpeg to generate a fixture and then spawn the FFX CLI as a subprocess.
The suite is integration-first and has almost no isolated unit-level coverage for pure logic.
The base Combinator class is a placeholder and is not the real abstraction boundary used by the suite.
Many combinator methods are placeholders: there are 25 pass statements across the current test modules.
Several assertion families are never executed because scenario selector dispatch is incomplete.
Scenario comments mention a Scenario 3, but no scenario_3.py exists.
tests/legacy/_basename_combinator_1.py is effectively orphaned because discovery only matches basename_combinator_*.py.
tests/legacy/disposition_combinator_2_3 .py contains an embedded space in the filename and is still part of discovery.
Expected failures are validated only as subprocess return-code matches, not as specific error types or messages.
The current suite depends on ffmpeg, ffprobe, SQLite, the local Python environment, and for Scenario 4 a live TMDB API key plus network access.

Rewrite Target

Replace the custom Click harness with a standard test runner, preferably pytest.
Split the suite into explicit layers: unit, integration, and optional external-system tests.
Keep unit tests as the default path and make them runnable without ffmpeg, ffprobe, TMDB, or a user config directory.
Model discovery explicitly in code instead of relying on glob-plus-reflection naming conventions.
Convert the current Cartesian-product combinators into readable parametrized cases grouped by behavior area.
Preserve the current behavior areas, but represent them with targeted cases instead of thousands of opaque variant IDs.
Make every assertion family explicit and executable; there must be no selector that is produced but never consumed.
Replace live TMDB access with fixtures or mocks in normal runs; any live-contract test must be opt-in.
Replace ad hoc subprocess return-code checks with assertions on typed exceptions, stderr content, or structured outputs.
Provide small reusable media fixtures or fixture builders so only a narrow integration slice needs ffmpeg-generated media.
Make database tests self-contained and fast through temporary databases and direct controller-level assertions.
Make ordering, naming, and selection deterministic so a contributor can predict exactly what will run.
Expose a small smoke suite for quick local runs and CI, plus a separately marked slower integration suite.
Prefer domain-oriented test modules over combinator-family modules: basename, pattern matching, metadata rewrite, track ordering, TMDB naming, CLI smoke, and failure handling.

Rewrite Acceptance

A default local test run finishes quickly and without network access.
A contributor can identify which behavior a failing test covers without decoding variant strings like VAASSS-A:D10-S:T001.
All current intended failure behaviors remain covered, but each one is asserted directly and readably.
The rewritten suite can be adopted by CI without requiring live TMDB credentials.

8.3 KiB Raw Blame History