ffx/SCRATCHPAD.md

Scratchpad

Goal

• Capture a compact, project-wide list of optimization candidates after a broad scan of the current FFX codebase, tooling, and requirements.

Settled

• The biggest near-term wins are in startup cost, repeated subprocess work, repeated database query patterns, and general repo hygiene.
• This list is intentionally optimization-oriented rather than bug-oriented. Some items below also improve correctness or maintainability, but they were selected because they can reduce runtime cost, operator friction, or iteration overhead.

Focused Snapshot

• Highest-leverage application optimizations:

  • Lazy-load CLI command dependencies so lightweight commands do not import most of the app.
  • Collapse repeated ffprobe calls into a single probe result per source file.
  • Replace query.count() plus first() patterns with single-query ORM accessors.
  • Cache or precompile filename pattern regexes instead of scanning every pattern for every file.
  • Guard logger handler installation to avoid duplicated handlers and noisy repeated setup.

• Highest-leverage repo and workflow optimizations:

  • Stop tracking nested __pycache__ output and other generated artifacts.
  • Consolidate setup and upgrade tooling to reduce overlapping shell-script responsibilities.
  • Trim or reorganize the oversized test/combinator surface so it is easier to run, debug, and extend.

Optimization Candidates

1. CLI startup and import cost

• src/ffx/ffx.py imports a large portion of the application at module import time, even for cheap commands such as version, help, setup_dependencies, and upgrade.
• Optimization:
  • Move heavy imports into the commands that actually need them.
  • Keep the CLI root importable with only core stdlib and Click dependencies.
• Expected value:
  • Faster startup for scripting and tooling commands.
  • Less coupling between maintenance commands and the runtime stack.

2. Repeated database queries via count() plus first()

• Controllers such as src/ffx/show_controller.py, src/ffx/pattern_controller.py, and src/ffx/database.py often do q.count() and then q.first().
• Optimization:
  • Replace with first(), one_or_none(), or existence checks that do not issue two queries.
  • Standardize this across all controllers.
• Expected value:
  • Lower SQLite query volume.
  • Simpler controller code.

3. Filename pattern matching scales linearly across all patterns

• src/ffx/pattern_controller.py loads every pattern and runs re.search against each filename on every lookup.
• Optimization:
  • Cache compiled regexes in process memory.
  • Stop after the first intentional match instead of silently returning the last match.
  • Consider explicit pattern priority if overlapping rules are valid.
• Expected value:
  • Faster per-file setup when many patterns exist.
  • More predictable matching behavior.

4. Media probing does two separate ffprobe subprocesses per file

• src/ffx/file_properties.py calls ffprobe once for format data and once for stream data.
• Optimization:
  • Use one probe call that requests both format and streams.
  • Cache that result inside FileProperties.
• Expected value:
  • Less subprocess overhead.
  • Faster inspect and convert flows.

5. Crop detection is always a full extra ffmpeg scan

• src/ffx/file_properties.py runs a dedicated ffmpeg -vf cropdetect pass for each file when crop detection is requested.
• Optimization:
  • Cache crop results for repeated runs on the same source.
  • Consider exposing shorter sampling windows or probe presets for large files.
• Expected value:
  • Lower latency on repeated experimentation.

6. Process wrapper lacks stronger execution controls

• src/ffx/process.py uses Popen(...).communicate() without timeout handling, structured error mapping, or direct missing-command handling.
• Optimization:
  • Add timeout support and clearer FileNotFoundError handling.
  • Consider subprocess.run(..., check=False, text=True) where streaming is not required.
  • Centralize return/error formatting.
• Expected value:
  • Better failure diagnosis.
  • Cleaner process management semantics.

7. Logger handlers can be added repeatedly

• src/ffx/ffx.py adds file and console handlers each invocation.
• Several helper classes install NullHandler instances ad hoc, for example src/ffx/process.py, src/ffx/tmdb_controller.py, src/ffx/media_descriptor.py, and src/ffx/helper.py.
• Optimization:
  • Guard handler installation so each logger is configured once.
  • Prefer module-level logger setup patterns over per-instance handler mutation.
• Expected value:
  • Less duplicate logging.
  • Lower confusion in long-running or repeatedly invoked contexts.

8. Repo-local hygiene for generated Python artifacts

• The repo currently contains nested compiled artifacts under src/ffx/__pycache__/....
• .gitignore only ignores __pycache__ at the repo root, not recursive __pycache__/.
• Optimization:
  • Ignore __pycache__/ recursively and clean tracked generated files.
  • Consider ignoring local virtualenv or other generated tool directories if they may appear in-repo later.
• Expected value:
  • Cleaner diffs and scans.
  • Lower repo noise.

9. Tooling overlap and naming drift

• There are now multiple prep-related scripts: tools/prepare.sh, tools/setup.sh, and the legacy-like tools/ffx_update.sh.
• Optimization:
  • Decide which scripts remain canonical.
  • Replace or remove legacy wrappers once equivalent CLI commands exist.
  • Keep CLI maintenance commands and shell wrappers aligned.
• Expected value:
  • Less operator confusion.
  • Fewer duplicated procedures to maintain.

10. Placeholder UI surfaces should either ship or disappear

• src/ffx/help_screen.py and src/ffx/settings_screen.py are placeholders.
• Optimization:
  • Either remove them from the active UI surface or complete them.
  • Avoid paying ongoing maintenance cost for unfinished navigation targets.
• Expected value:
  • Leaner interface.
  • Lower UX ambiguity.

11. Large Textual screens repeat configuration and controller loading

• Screens such as src/ffx/media_details_screen.py, src/ffx/pattern_details_screen.py, and src/ffx/show_details_screen.py repeat setup patterns and local metadata filtering extraction.
• Optimization:
  • Extract a shared screen base or helper for common config/controller/bootstrap logic.
  • Reduce repeated table refresh and repeated DB fetch code where possible.
• Expected value:
  • Lower maintenance overhead.
  • Easier UI iteration.

12. Several helper functions are unfinished or dead-weight

• src/ffx/helper.py contains permutateList(...): pass.
• There are many combinator and conversion placeholders across tests and migrations.
• Optimization:
  • Remove dead code, finish it, or isolate it behind a clearly dormant area.
  • Avoid carrying stubbed utility surface that looks reusable but is not.
• Expected value:
  • Smaller mental model.
  • Less time spent re-evaluating inactive paths.

13. Test suite shape is expensive to understand and likely expensive to run

• The project has a large matrix of combinator files under src/ffx/test, several placeholder pass implementations, and at least one suspicious filename with an embedded space: [src/ffx/test/disposition_combinator_2_3 .py](/home/osgw/.local/src/codex/ffx/src/ffx/test/disposition_combinator_2_3 .py).
• Optimization:
  • Consolidate combinator families.
  • Add a lighter smoke-test path.
  • Normalize file naming and test discovery conventions.
• Expected value:
  • Faster contributor onboarding.
  • Easier CI adoption later.

14. Process resource limiting semantics could be clearer

• src/ffx/process.py prepends nice and cpulimit directly when values are set.
• Optimization:
  • Validate and document effective behavior for combined nice + cpulimit.
  • Consider explicit no-limit vs configured-limit states in the CLI and requirements.
• Expected value:
  • Fewer surprises in production-like runs.
  • Easier support for user-reported performance behavior.

15. Import-time dependency coupling makes maintenance commands brittle

• Even after recent CLI maintenance additions, the top-level CLI module still imports most application modules before Click dispatch.
• Optimization:
  • Push imports for ORM, Textual, TMDB, ffmpeg helpers, and descriptors behind the commands that actually need them.
• Expected value:
  • Maintenance commands such as setup and upgrade stay usable when optional runtime dependencies are broken.
  • Better separation between media runtime code and maintenance tooling.

16. Regex and string utility cleanup

• src/ffx/helper.py still emits a SyntaxWarning for RICH_COLOR_PATTERN.
• Optimization:
  • Convert regex literals to raw strings where appropriate.
  • Review filename and TMDB substitution helpers for repeated string churn.
• Expected value:
  • Cleaner runtime output.
  • Less warning noise during dry-run maintenance commands.

17. Database startup always runs schema creation and version checks

• src/ffx/database.py runs Base.metadata.create_all(...) and version checks every time a DB-backed context is created.
• Optimization:
  • Measure startup cost and consider separating bootstrapping from ordinary command execution.
  • Keep schema migration/version enforcement explicit.
• Expected value:
  • Faster command startup.
  • Clearer operational boundaries.

Open

• Should optimization work focus first on operator-perceived latency, internal maintainability, or correctness-risk cleanup that also has performance upside?
• Is the long-term supported model still “local Linux workstation plus Textual UI,” or should optimization decisions bias toward a more scriptable/headless CLI?

Gaps Right Now

• No explicit prioritization owner or milestone for the optimization backlog.
• No benchmark or timing harness exists for startup, probe, DB, or conversion orchestration overhead.
• Repo hygiene is still mixed with generated artifacts and some clearly unfinished files.

Next

1. Triage the list into quick wins, medium refactors, and long-horizon cleanup.
2. Tackle the cheapest high-impact items first:
   • recursive __pycache__/ ignore and cleanup,
   • regex raw-string warning cleanup,
   • count() plus first() query cleanup,
   • single-call ffprobe refactor.
3. Decide whether maintenance/tooling command imports should be split from media-runtime imports before adding more CLI maintenance surface.

Delete When

• Delete this scratchpad once the optimization backlog is either converted into issues/work items or distilled into durable project guidance.