ffx/requirements/tests.md

Test Rewrite

This file captures the structure executed by tests/legacy_runner.py today and defines the target shape for a complete rewrite.

Detailed product rules for source-to-target subtrack mapping live in requirements/subtrack_mapping.md. This file describes only how tests cover that area.

Interpreter Requirement

• Agents shall run Python-side test commands with ~/.local/share/ffx.venv/bin/python.
• This applies to the legacy harness, unittest, pytest, helper scripts, and python -m ffx ... test invocations.
• Agents shall not silently substitute python, python3, or another interpreter for Python-side test work.
• If ~/.local/share/ffx.venv/bin/python is missing or not executable, agents shall stop and report the missing venv instead of continuing with Python-side test execution.

Current Harness

• Entrypoint: ~/.local/share/ffx.venv/bin/python tests/legacy_runner.py run
• Runner style: custom Click CLI, not pytest or unittest
• Commands:
  • run: discover scenario files, instantiate each scenario, run yielded jobs
  • dupe: helper command that creates duplicate media fixtures; not part of the test run
• Filters: --scenario, --variant, --limit
• Shared context:
  • builds one mutable dict for the whole run
  • installs loggers and writes ffx_test_report.log
  • creates ConfigurationController eagerly
  • tracks only passed and failed counters
• Discovery:
  • scenario files: tests/legacy/scenario_*.py
  • combinators: glob + importlib + inspect by filename convention
  • ordering: implicit glob order, no explicit sorting
• Skip behavior:
  • Scenario 4 is skipped when TMDB_API_KEY is missing
  • only TMDB_API_KEY_NOT_PRESENT_EXCEPTION is caught at scenario construction time

Current Scenarios

• 1: tests/legacy/scenario_1.py
  • focus: basename generation without pattern lookup or TMDB
  • inputs per job: 1
  • jobs: 140
  • expected failures: 0
  • execution: build one synthetic source file, run ~/.local/share/ffx.venv/bin/python -m ffx convert, assert filename selectors only
  • selectors executed: B, L, I
  • selectors defined but not executed: S, R
• 2: tests/legacy/scenario_2.py
  • focus: conversion matrix over media layouts, dispositions, tags, and permutations
  • inputs per job: 1
  • jobs: 8193
  • expected failures: 3267
  • execution: build one synthetic source file, run ~/.local/share/ffx.venv/bin/python -m ffx convert, probe result with FileProperties, assert track layout and selected audio and subtitle metadata
  • selectors executed: M, AD, AT, SD, ST
  • selectors defined but not executed: MT, AP, SP, J
• 4: tests/legacy/scenario_4.py
  • focus: pattern-driven batch conversion with SQLite state and live TMDB naming
  • inputs per job: 6
  • jobs: 768
  • expected failures: 336
  • execution: build six synthetic preset files, recreate temp SQLite DB, insert show and pattern, run one batch convert command via ~/.local/share/ffx.venv/bin/python, query TMDB during assertions
  • selectors executed: M, AD, AT, SD, ST
  • selectors defined but not executed: MT, AP, SP, J
  • notes:
    • uses MediaCombinator6 only
    • issues live HTTP requests through TmdbController with no request cache

Current Combinator Families

• scenario files discovered: 3
• basename combinators discovered: 2
• media combinators discovered: 8
• media tag combinators discovered: 3
• disposition combinator 2 variants: 4
• disposition combinator 3 variants: 5
• track tag combinator 2 variants: 4
• track tag combinator 3 variants: 5
• indicator variants: 7
• label variants: 2
• show variants: 3
• release variants: 3
• permutation 2 variants: 2
• permutation 3 variants: 3

Current Totals

• full run without TMDB: 8333
• full run with TMDB: 9101
• Scenario 4 generated source files: 4608
• Scenario 4 live TMDB episode queries: 4608

Current Behavior Areas

• output basename rules for label, season and episode indicator, show name, and release suffix combinations
• track layout normalization across the eight media combinator shapes from VA through VAASSS
• two-track and three-track disposition edge cases, including intentional failure cases
• two-track and three-track track-tag preservation checks, including checks that sort results by source identity
• container-level media tag handling
• pattern-backed conversion against a temporary SQLite database
• TMDB-assisted episode naming for batch conversion

Structural Findings

• The suite is process-heavy: most jobs run ffmpeg to generate a fixture and then spawn the FFX CLI as a subprocess.
• The suite is integration-first and has almost no isolated unit-level coverage for pure logic.
• The base Combinator class is a placeholder and is not the real abstraction boundary used by the suite.
• Many combinator methods are placeholders: there are 25 pass statements across the current test modules.
• Several assertion families are never executed because scenario selector dispatch is incomplete.
• Scenario comments mention a Scenario 3, but no scenario_3.py exists.
• tests/legacy/_basename_combinator_1.py is effectively orphaned because discovery only matches basename_combinator_*.py.
• tests/legacy/disposition_combinator_2_3 .py contains an embedded space in the filename and is still part of discovery.
• Expected failures are validated only as subprocess return-code matches, not as specific error types or messages.
• The current suite depends on ffmpeg, ffprobe, SQLite, the local Python environment, and for Scenario 4 a live TMDB API key plus network access.

Rewrite Target

• Replace the custom Click harness with a standard test runner, preferably pytest.
• Split the suite into explicit layers: unit, integration, and optional external-system tests.
• Keep unit tests as the default path and make them runnable without ffmpeg, ffprobe, TMDB, or a user config directory.
• Model discovery explicitly in code instead of relying on glob-plus-reflection naming conventions.
• Convert the current Cartesian-product combinators into readable parametrized cases grouped by behavior area.
• Preserve the current behavior areas, but represent them with targeted cases instead of thousands of opaque variant IDs.
• Make every assertion family explicit and executable; there must be no selector that is produced but never consumed.
• Replace live TMDB access with fixtures or mocks in normal runs; any live-contract test must be opt-in.
• Replace ad hoc subprocess return-code checks with assertions on typed exceptions, stderr content, or structured outputs.
• Provide small reusable media fixtures or fixture builders so only a narrow integration slice needs ffmpeg-generated media.
• Make database tests self-contained and fast through temporary databases and direct controller-level assertions.
• Make ordering, naming, and selection deterministic so a contributor can predict exactly what will run.
• Expose a small smoke suite for quick local runs and CI, plus a separately marked slower integration suite.
• Prefer domain-oriented test modules over combinator-family modules: basename, pattern matching, metadata rewrite, track ordering, TMDB naming, CLI smoke, and failure handling.

Rewrite Acceptance

• A default local test run finishes quickly and without network access.
• A contributor can identify which behavior a failing test covers without decoding variant strings like VAASSS-A:D10-S:T001.
• All current intended failure behaviors remain covered, but each one is asserted directly and readably.
• The rewritten suite can be adopted by CI without requiring live TMDB credentials.