diff --git a/docs/superpowers/plans/2026-08-23-phase1-foundation.md b/docs/superpowers/plans/2026-08-23-phase1-foundation.md new file mode 100644 index 0000000..c9bdc0b --- /dev/null +++ b/docs/superpowers/plans/2026-08-23-phase1-foundation.md @@ -0,0 +1,1275 @@ +# Phase 1: Foundation — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Stand up the `agent` worker container — config, `agentdb` with migrations, a read-only Firefly API client, a scheduled heartbeat that proves end-to-end connectivity — and deploy it to the production host. + +**Architecture:** A Python 3.12 package under `agent/` in this repo, run as one container in the existing compose stack. On startup it creates/migrates `agentdb` (a second database in the existing Postgres container), runs a heartbeat once, then hands off to APScheduler. The heartbeat calls Firefly's `/api/v1/about` and records the outcome in `sync_runs` — the same logging path every future connector will use. + +**Tech Stack:** Python 3.12, APScheduler 3.x, httpx, psycopg 3, pytest, testcontainers (dev only), Docker/compose. + +**Spec:** `docs/superpowers/specs/2026-08-23-accounting-agent-architecture-design.md` + +## Global Constraints + +- Python 3.12+ (`requires-python = ">=3.12"`); container base image `python:3.12-slim`. +- Runtime dependencies limited to: `apscheduler>=3.10,<4`, `httpx>=0.27`, `psycopg[binary]>=3.1`. Dev-only: `pytest>=8`, `testcontainers[postgres]>=4`. +- TDD: every code task writes the failing test first, sees it fail, then implements. +- Firefly API access in this phase is **read-only** (GET endpoints only). The agent never writes to the ledger. +- Secrets live only in `.env` on the host. Never commit `.env`, tokens, or passwords. +- All timestamps are UTC (`TIMESTAMPTZ` in Postgres, `datetime.now(timezone.utc)` in Python). +- The `agent` container attaches only to the internal `firefly` network: no Traefik labels, no published ports. +- Commit after every green test cycle. +- Prerequisite for running tests locally: Docker daemon running (testcontainers starts a throwaway `postgres:16-alpine`). +- Testing deviation from spec, by design: the spec's throwaway-Firefly compose test profile arrives in Phase 2 with the sync engine. Phase 1's client makes two GET calls, tested against `httpx.MockTransport`; Task 9's `agent.verify` exercises the real API in production. + +## File Structure + +``` +agent/ +├── Dockerfile +├── .dockerignore +├── pyproject.toml +├── README.md # setup + deploy runbook (Task 8) +├── src/agent/ +│ ├── __init__.py # __version__ +│ ├── __main__.py # entrypoint: ensure db → migrate → heartbeat → scheduler +│ ├── config.py # AgentConfig, load_config, ConfigError +│ ├── db.py # DSNs, ensure_database, run_migrations, connect +│ ├── migrations/0001_initial.sql +│ ├── firefly.py # FireflyClient (read-only), FireflyError +│ ├── scheduler.py # build_scheduler +│ ├── healthcheck.py # Docker HEALTHCHECK entry +│ ├── verify.py # `python -m agent.verify` post-deploy check +│ └── jobs/ +│ ├── __init__.py +│ └── heartbeat.py # run_heartbeat +└── tests/ + ├── conftest.py # session postgres container, per-test cfg + ├── test_smoke.py + ├── test_config.py + ├── test_db.py + ├── test_firefly.py + ├── test_heartbeat.py + ├── test_scheduler.py + └── test_verify.py +``` + +Everything runs from `agent/` (`cd agent && pytest`). Repo-root files touched: `docker-compose.yml`, `.env.example`, `.gitignore`. + +--- + +### Task 1: Package scaffold + +**Files:** +- Create: `agent/pyproject.toml`, `agent/src/agent/__init__.py`, `agent/src/agent/jobs/__init__.py`, `agent/tests/test_smoke.py` +- Modify: `.gitignore` + +**Interfaces:** +- Produces: importable package `agent` with `agent.__version__ == "0.1.0"`; working `pytest` invocation from `agent/`. + +- [ ] **Step 1: Create `agent/pyproject.toml`** + +```toml +[project] +name = "firefly-agent" +version = "0.1.0" +description = "Personal accounting agent worker for the Firefly III stack" +requires-python = ">=3.12" +dependencies = [ + "apscheduler>=3.10,<4", + "httpx>=0.27", + "psycopg[binary]>=3.1", +] + +[project.optional-dependencies] +dev = [ + "pytest>=8", + "testcontainers[postgres]>=4", +] + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.hatch.build.targets.wheel] +packages = ["src/agent"] + +[tool.pytest.ini_options] +testpaths = ["tests"] +``` + +- [ ] **Step 2: Write the failing smoke test** — `agent/tests/test_smoke.py` + +```python +def test_package_imports(): + import agent + + assert agent.__version__ == "0.1.0" +``` + +- [ ] **Step 3: Set up the venv and run the test to verify it fails** + +```bash +cd /home/jake/dev/firefly +python3 -m venv .venv +source .venv/bin/activate +pip install -e './agent[dev]' +cd agent && pytest tests/test_smoke.py -v +``` + +Expected: FAIL (`agent` has no attribute `__version__` — the editable install will fail or the import will, since `src/agent/__init__.py` doesn't exist yet; create empty dirs as needed to get the install through, then see the assertion fail). + +- [ ] **Step 4: Implement** — `agent/src/agent/__init__.py` + +```python +__version__ = "0.1.0" +``` + +Also create empty `agent/src/agent/jobs/__init__.py`. + +- [ ] **Step 5: Run test to verify it passes** + +Run: `cd agent && pytest tests/test_smoke.py -v` — Expected: PASS + +- [ ] **Step 6: Add ignores to repo-root `.gitignore`** + +Append these lines: + +``` +.venv/ +__pycache__/ +*.egg-info/ +.pytest_cache/ +``` + +- [ ] **Step 7: Commit** + +```bash +git add agent/ .gitignore +git commit -m "feat(agent): scaffold Python package for the accounting agent worker" +``` + +--- + +### Task 2: Config loading + +**Files:** +- Create: `agent/src/agent/config.py` +- Test: `agent/tests/test_config.py` + +**Interfaces:** +- Produces: + - `class ConfigError(ValueError)` + - `@dataclass(frozen=True) AgentConfig` with fields: `db_host: str`, `db_port: int`, `db_user: str`, `db_password: str`, `agent_db_name: str`, `firefly_url: str`, `firefly_token: str`, `heartbeat_interval_minutes: int` + - `load_config(env: Mapping[str, str] | None = None) -> AgentConfig` — reads `os.environ` when `env` is None. + +Environment variables (shared names match the existing `.env`): required — `DB_HOST`, `POSTGRES_USER`, `POSTGRES_PASSWORD`, `FIREFLY_API_URL`, `AGENT_FIREFLY_TOKEN`; optional with defaults — `DB_PORT`=5432, `AGENT_DB_NAME`=`agentdb`, `HEARTBEAT_INTERVAL_MINUTES`=60. + +- [ ] **Step 1: Write the failing tests** — `agent/tests/test_config.py` + +```python +import pytest + +from agent.config import AgentConfig, ConfigError, load_config + +FULL_ENV = { + "DB_HOST": "db", + "DB_PORT": "5433", + "POSTGRES_USER": "firefly", + "POSTGRES_PASSWORD": "s3cret", + "AGENT_DB_NAME": "agentdb2", + "FIREFLY_API_URL": "http://app:8080", + "AGENT_FIREFLY_TOKEN": "tok123", + "HEARTBEAT_INTERVAL_MINUTES": "15", +} + +MINIMAL_ENV = { + "DB_HOST": "db", + "POSTGRES_USER": "firefly", + "POSTGRES_PASSWORD": "s3cret", + "FIREFLY_API_URL": "http://app:8080", + "AGENT_FIREFLY_TOKEN": "tok123", +} + + +def test_load_config_reads_all_vars(): + cfg = load_config(FULL_ENV) + assert cfg == AgentConfig( + db_host="db", + db_port=5433, + db_user="firefly", + db_password="s3cret", + agent_db_name="agentdb2", + firefly_url="http://app:8080", + firefly_token="tok123", + heartbeat_interval_minutes=15, + ) + + +def test_defaults_applied(): + cfg = load_config(MINIMAL_ENV) + assert cfg.db_port == 5432 + assert cfg.agent_db_name == "agentdb" + assert cfg.heartbeat_interval_minutes == 60 + + +@pytest.mark.parametrize( + "missing", + ["DB_HOST", "POSTGRES_USER", "POSTGRES_PASSWORD", "FIREFLY_API_URL", "AGENT_FIREFLY_TOKEN"], +) +def test_missing_required_var_names_the_var(missing): + env = {k: v for k, v in MINIMAL_ENV.items() if k != missing} + with pytest.raises(ConfigError, match=missing): + load_config(env) +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `cd agent && pytest tests/test_config.py -v` — Expected: FAIL with `ModuleNotFoundError: No module named 'agent.config'` + +- [ ] **Step 3: Implement** — `agent/src/agent/config.py` + +```python +"""Agent configuration loaded from environment variables.""" + +from __future__ import annotations + +import os +from collections.abc import Mapping +from dataclasses import dataclass + + +class ConfigError(ValueError): + """A required environment variable is missing or invalid.""" + + +@dataclass(frozen=True) +class AgentConfig: + db_host: str + db_port: int + db_user: str + db_password: str + agent_db_name: str + firefly_url: str + firefly_token: str + heartbeat_interval_minutes: int + + +def _require(env: Mapping[str, str], name: str) -> str: + value = env.get(name, "").strip() + if not value: + raise ConfigError(f"missing required environment variable: {name}") + return value + + +def load_config(env: Mapping[str, str] | None = None) -> AgentConfig: + if env is None: + env = os.environ + return AgentConfig( + db_host=_require(env, "DB_HOST"), + db_port=int(env.get("DB_PORT", "5432")), + db_user=_require(env, "POSTGRES_USER"), + db_password=_require(env, "POSTGRES_PASSWORD"), + agent_db_name=env.get("AGENT_DB_NAME", "agentdb"), + firefly_url=_require(env, "FIREFLY_API_URL"), + firefly_token=_require(env, "AGENT_FIREFLY_TOKEN"), + heartbeat_interval_minutes=int(env.get("HEARTBEAT_INTERVAL_MINUTES", "60")), + ) +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `cd agent && pytest tests/test_config.py -v` — Expected: PASS (7 tests) + +- [ ] **Step 5: Commit** + +```bash +git add agent/src/agent/config.py agent/tests/test_config.py +git commit -m "feat(agent): load config from environment with validation" +``` + +--- + +### Task 3: agentdb — database bootstrap and migrations + +**Files:** +- Create: `agent/src/agent/db.py`, `agent/src/agent/migrations/0001_initial.sql`, `agent/tests/conftest.py` +- Test: `agent/tests/test_db.py` + +**Interfaces:** +- Consumes: `AgentConfig` from Task 2. +- Produces: + - `admin_dsn(cfg: AgentConfig) -> str` — DSN to the always-present `postgres` maintenance database. + - `agent_dsn(cfg: AgentConfig) -> str` — DSN to `cfg.agent_db_name`. + - `ensure_database(cfg: AgentConfig) -> None` — creates `agentdb` if absent; idempotent. + - `run_migrations(cfg: AgentConfig) -> list[str]` — applies pending `migrations/*.sql` in name order, returns the names applied (empty list when up to date). + - `connect(cfg: AgentConfig) -> psycopg.Connection` — connection to agentdb. + - Tables after migration 0001: `schema_migrations`, `sync_runs`, `documents`, `bills`, `approvals`. +- Test fixtures produced for later tasks (in `conftest.py`): `pg` (session-scoped Postgres container), `cfg` (per-test `AgentConfig` pointing at a unique database name in that container). + +- [ ] **Step 1: Write the migration SQL** — `agent/src/agent/migrations/0001_initial.sql` + +This is data, not logic — write it before the tests that assert on it. Schema is the spec's "agentdb schema (v1)" verbatim: + +```sql +-- agentdb schema v1 (see spec: agentdb schema section) + +CREATE TABLE sync_runs ( + id BIGSERIAL PRIMARY KEY, + connector TEXT NOT NULL, + started_at TIMESTAMPTZ NOT NULL, + finished_at TIMESTAMPTZ, + status TEXT NOT NULL CHECK (status IN ('running', 'ok', 'error')), + items_upserted INTEGER NOT NULL DEFAULT 0, + detail TEXT +); + +CREATE INDEX sync_runs_connector_id_idx ON sync_runs (connector, id DESC); + +CREATE TABLE documents ( + id BIGSERIAL PRIMARY KEY, + path TEXT NOT NULL UNIQUE, + sha256 CHAR(64) NOT NULL UNIQUE, + institution TEXT NOT NULL, + account TEXT, + doc_type TEXT NOT NULL, + period TEXT, + source TEXT NOT NULL, + received_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +CREATE TABLE bills ( + id BIGSERIAL PRIMARY KEY, + payee TEXT NOT NULL, + expected_amount NUMERIC(12,2), + due_date DATE NOT NULL, + funding_account TEXT, + autopay BOOLEAN NOT NULL DEFAULT false, + status TEXT NOT NULL DEFAULT 'upcoming' + CHECK (status IN ('upcoming', 'approved', 'paid', 'skipped', 'overdue')) +); + +CREATE TABLE approvals ( + id BIGSERIAL PRIMARY KEY, + bill_id BIGINT NOT NULL REFERENCES bills (id), + amount NUMERIC(12,2) NOT NULL, + requested_at TIMESTAMPTZ NOT NULL DEFAULT now(), + decided_at TIMESTAMPTZ, + decision TEXT CHECK (decision IN ('approved', 'denied')), + telegram_message_id TEXT, + executed_at TIMESTAMPTZ +); +``` + +(`schema_migrations` is created by the runner itself, not by a migration.) + +- [ ] **Step 2: Write the shared test fixtures** — `agent/tests/conftest.py` + +```python +import itertools + +import pytest +from testcontainers.postgres import PostgresContainer + +from agent.config import AgentConfig + +_db_counter = itertools.count() + + +@pytest.fixture(scope="session") +def pg(): + with PostgresContainer("postgres:16-alpine") as container: + yield container + + +@pytest.fixture +def cfg(pg) -> AgentConfig: + """AgentConfig pointing at a fresh, uniquely named database per test.""" + return AgentConfig( + db_host=pg.get_container_host_ip(), + db_port=int(pg.get_exposed_port(5432)), + db_user=pg.username, + db_password=pg.password, + agent_db_name=f"agentdb_test_{next(_db_counter)}", + firefly_url="http://firefly.test", + firefly_token="test-token", + heartbeat_interval_minutes=60, + ) +``` + +- [ ] **Step 3: Write the failing tests** — `agent/tests/test_db.py` + +```python +from agent.db import connect, ensure_database, run_migrations + +EXPECTED_TABLES = {"schema_migrations", "sync_runs", "documents", "bills", "approvals"} + + +def test_ensure_database_creates_and_is_idempotent(cfg): + ensure_database(cfg) + ensure_database(cfg) # second call must not raise + with connect(cfg) as conn: + assert conn.execute("SELECT 1").fetchone()[0] == 1 + + +def test_run_migrations_applies_each_migration_exactly_once(cfg): + ensure_database(cfg) + + first = run_migrations(cfg) + assert first == ["0001_initial.sql"] + + second = run_migrations(cfg) + assert second == [] + + with connect(cfg) as conn: + rows = conn.execute( + "SELECT tablename FROM pg_tables WHERE schemaname = 'public'" + ).fetchall() + assert EXPECTED_TABLES <= {r[0] for r in rows} + + +def test_sync_runs_rejects_bad_status(cfg): + import psycopg + import pytest + + ensure_database(cfg) + run_migrations(cfg) + with connect(cfg) as conn, pytest.raises(psycopg.errors.CheckViolation): + conn.execute( + "INSERT INTO sync_runs (connector, started_at, status)" + " VALUES ('x', now(), 'bogus')" + ) +``` + +- [ ] **Step 4: Run tests to verify they fail** + +Run: `cd agent && pytest tests/test_db.py -v` — Expected: FAIL with `ModuleNotFoundError: No module named 'agent.db'` (the container may take ~10s to start first run) + +- [ ] **Step 5: Implement** — `agent/src/agent/db.py` + +```python +"""agentdb access: bootstrap, migrations, connections.""" + +from __future__ import annotations + +from pathlib import Path + +import psycopg + +from agent.config import AgentConfig + +_MIGRATIONS_DIR = Path(__file__).parent / "migrations" + + +def _dsn(cfg: AgentConfig, dbname: str) -> str: + return ( + f"host={cfg.db_host} port={cfg.db_port} dbname={dbname} " + f"user={cfg.db_user} password={cfg.db_password}" + ) + + +def admin_dsn(cfg: AgentConfig) -> str: + """DSN for the maintenance database that always exists.""" + return _dsn(cfg, "postgres") + + +def agent_dsn(cfg: AgentConfig) -> str: + return _dsn(cfg, cfg.agent_db_name) + + +def connect(cfg: AgentConfig) -> psycopg.Connection: + return psycopg.connect(agent_dsn(cfg)) + + +def ensure_database(cfg: AgentConfig) -> None: + """Create the agent database if it does not exist. Idempotent.""" + with psycopg.connect(admin_dsn(cfg), autocommit=True) as conn: + exists = conn.execute( + "SELECT 1 FROM pg_database WHERE datname = %s", (cfg.agent_db_name,) + ).fetchone() + if not exists: + # CREATE DATABASE cannot be parameterized; the name comes from + # our own config, quoted defensively via psycopg's identifier API. + from psycopg import sql + + conn.execute( + sql.SQL("CREATE DATABASE {}").format(sql.Identifier(cfg.agent_db_name)) + ) + + +def run_migrations(cfg: AgentConfig) -> list[str]: + """Apply pending migrations in filename order. Returns names applied.""" + applied: list[str] = [] + with connect(cfg) as conn: + conn.execute( + "CREATE TABLE IF NOT EXISTS schema_migrations (" + " name TEXT PRIMARY KEY," + " applied_at TIMESTAMPTZ NOT NULL DEFAULT now())" + ) + done = { + row[0] for row in conn.execute("SELECT name FROM schema_migrations") + } + for path in sorted(_MIGRATIONS_DIR.glob("*.sql")): + if path.name in done: + continue + conn.execute(path.read_text()) + conn.execute( + "INSERT INTO schema_migrations (name) VALUES (%s)", (path.name,) + ) + applied.append(path.name) + return applied +``` + +- [ ] **Step 6: Run tests to verify they pass** + +Run: `cd agent && pytest tests/test_db.py -v` — Expected: PASS (3 tests) + +- [ ] **Step 7: Commit** + +```bash +git add agent/src/agent/db.py agent/src/agent/migrations/ agent/tests/conftest.py agent/tests/test_db.py +git commit -m "feat(agent): agentdb bootstrap, migration runner, v1 schema" +``` + +--- + +### Task 4: Firefly API client (read-only) + +**Files:** +- Create: `agent/src/agent/firefly.py` +- Test: `agent/tests/test_firefly.py` + +**Interfaces:** +- Produces: + - `class FireflyError(Exception)` + - `class FireflyClient` with: + - `__init__(self, base_url: str, token: str, transport: httpx.BaseTransport | None = None)` — `transport` exists for tests only. + - `about(self) -> dict` — returns the `data` object from `GET /api/v1/about` (contains `version`). + - `accounts(self, type: str = "asset") -> list[dict]` — all pages of `GET /api/v1/accounts`; each item is Firefly's account object (`{"id": ..., "attributes": {"name": ...}}`). + +- [ ] **Step 1: Write the failing tests** — `agent/tests/test_firefly.py` + +```python +import httpx +import pytest + +from agent.firefly import FireflyClient, FireflyError + + +def make_client(handler) -> FireflyClient: + return FireflyClient( + "http://firefly.test", "tok", transport=httpx.MockTransport(handler) + ) + + +def test_about_sends_bearer_token_and_returns_data(): + def handler(request: httpx.Request) -> httpx.Response: + assert request.headers["Authorization"] == "Bearer tok" + assert request.headers["Accept"] == "application/json" + assert request.url.path == "/api/v1/about" + return httpx.Response(200, json={"data": {"version": "6.1.1"}}) + + assert make_client(handler).about() == {"version": "6.1.1"} + + +def test_non_200_raises_firefly_error_with_status(): + def handler(request: httpx.Request) -> httpx.Response: + return httpx.Response(401, json={"message": "Unauthenticated"}) + + with pytest.raises(FireflyError, match="401"): + make_client(handler).about() + + +def test_accounts_walks_all_pages(): + def handler(request: httpx.Request) -> httpx.Response: + assert request.url.path == "/api/v1/accounts" + assert request.url.params["type"] == "asset" + page = int(request.url.params.get("page", "1")) + data = { + 1: [{"id": "1", "attributes": {"name": "NFCU Checking"}}], + 2: [{"id": "2", "attributes": {"name": "Venmo"}}], + }[page] + return httpx.Response( + 200, + json={ + "data": data, + "meta": {"pagination": {"total_pages": 2, "current_page": page}}, + }, + ) + + accounts = make_client(handler).accounts() + assert [a["attributes"]["name"] for a in accounts] == ["NFCU Checking", "Venmo"] +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `cd agent && pytest tests/test_firefly.py -v` — Expected: FAIL with `ModuleNotFoundError: No module named 'agent.firefly'` + +- [ ] **Step 3: Implement** — `agent/src/agent/firefly.py` + +```python +"""Minimal read-only client for the Firefly III REST API.""" + +from __future__ import annotations + +import httpx + + +class FireflyError(Exception): + """Firefly III returned an unexpected response.""" + + +class FireflyClient: + def __init__( + self, + base_url: str, + token: str, + transport: httpx.BaseTransport | None = None, + ) -> None: + self._http = httpx.Client( + base_url=base_url, + headers={ + "Authorization": f"Bearer {token}", + "Accept": "application/json", + }, + timeout=30.0, + transport=transport, + ) + + def _get(self, path: str, params: dict | None = None) -> dict: + response = self._http.get(path, params=params) + if response.status_code != 200: + raise FireflyError( + f"GET {path} -> {response.status_code}: {response.text[:200]}" + ) + return response.json() + + def about(self) -> dict: + return self._get("/api/v1/about")["data"] + + def accounts(self, type: str = "asset") -> list[dict]: + results: list[dict] = [] + page = 1 + while True: + body = self._get("/api/v1/accounts", params={"type": type, "page": page}) + results.extend(body["data"]) + pagination = body.get("meta", {}).get("pagination", {}) + if page >= pagination.get("total_pages", 1): + return results + page += 1 +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `cd agent && pytest tests/test_firefly.py -v` — Expected: PASS (3 tests) + +- [ ] **Step 5: Commit** + +```bash +git add agent/src/agent/firefly.py agent/tests/test_firefly.py +git commit -m "feat(agent): read-only Firefly III API client" +``` + +--- + +### Task 5: Heartbeat job + +**Files:** +- Create: `agent/src/agent/jobs/heartbeat.py` +- Test: `agent/tests/test_heartbeat.py` + +**Interfaces:** +- Consumes: `AgentConfig` (Task 2), `connect` (Task 3), `FireflyClient`/`FireflyError` (Task 4). +- Produces: `run_heartbeat(cfg: AgentConfig, client: FireflyClient | None = None) -> str` — calls Firefly `about()`, inserts one `sync_runs` row (`connector='heartbeat'`, status `'ok'` or `'error'`), returns the status string. Builds its own client from `cfg` when `client` is None; the parameter exists for tests. + +- [ ] **Step 1: Write the failing tests** — `agent/tests/test_heartbeat.py` + +```python +from agent.db import connect, ensure_database, run_migrations +from agent.firefly import FireflyError +from agent.jobs.heartbeat import run_heartbeat + + +class FakeFirefly: + def __init__(self, version: str | None = None, error: Exception | None = None): + self._version = version + self._error = error + + def about(self) -> dict: + if self._error: + raise self._error + return {"version": self._version} + + +def _last_run(cfg): + with connect(cfg) as conn: + return conn.execute( + "SELECT connector, status, detail, started_at, finished_at" + " FROM sync_runs ORDER BY id DESC LIMIT 1" + ).fetchone() + + +def test_heartbeat_records_ok(cfg): + ensure_database(cfg) + run_migrations(cfg) + + status = run_heartbeat(cfg, client=FakeFirefly(version="6.1.1")) + + assert status == "ok" + connector, db_status, detail, started_at, finished_at = _last_run(cfg) + assert connector == "heartbeat" + assert db_status == "ok" + assert "6.1.1" in detail + assert finished_at >= started_at + + +def test_heartbeat_records_error_and_does_not_raise(cfg): + ensure_database(cfg) + run_migrations(cfg) + + status = run_heartbeat(cfg, client=FakeFirefly(error=FireflyError("GET /api/v1/about -> 401"))) + + assert status == "error" + connector, db_status, detail, *_ = _last_run(cfg) + assert db_status == "error" + assert "401" in detail +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `cd agent && pytest tests/test_heartbeat.py -v` — Expected: FAIL with `ModuleNotFoundError: No module named 'agent.jobs.heartbeat'` + +- [ ] **Step 3: Implement** — `agent/src/agent/jobs/heartbeat.py` + +```python +"""Heartbeat: prove Firefly API + agentdb connectivity on a schedule.""" + +from __future__ import annotations + +import logging +from datetime import datetime, timezone + +from agent.config import AgentConfig +from agent.db import connect +from agent.firefly import FireflyClient + +logger = logging.getLogger(__name__) + + +def run_heartbeat(cfg: AgentConfig, client: FireflyClient | None = None) -> str: + if client is None: + client = FireflyClient(cfg.firefly_url, cfg.firefly_token) + + started_at = datetime.now(timezone.utc) + try: + info = client.about() + status, detail = "ok", f"firefly version {info.get('version', 'unknown')}" + except Exception as exc: # any failure is a recorded outcome, not a crash + status, detail = "error", f"{type(exc).__name__}: {exc}" + logger.warning("heartbeat failed: %s", detail) + + finished_at = datetime.now(timezone.utc) + with connect(cfg) as conn: + conn.execute( + "INSERT INTO sync_runs (connector, started_at, finished_at, status, detail)" + " VALUES (%s, %s, %s, %s, %s)", + ("heartbeat", started_at, finished_at, status, detail), + ) + return status +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `cd agent && pytest tests/test_heartbeat.py -v` — Expected: PASS (2 tests) + +- [ ] **Step 5: Commit** + +```bash +git add agent/src/agent/jobs/heartbeat.py agent/tests/test_heartbeat.py +git commit -m "feat(agent): heartbeat job recording to sync_runs" +``` + +--- + +### Task 6: Scheduler, healthcheck, entrypoint + +**Files:** +- Create: `agent/src/agent/scheduler.py`, `agent/src/agent/healthcheck.py`, `agent/src/agent/__main__.py` +- Test: `agent/tests/test_scheduler.py` + +**Interfaces:** +- Consumes: `AgentConfig`, `ensure_database`, `run_migrations`, `connect`, `run_heartbeat`. +- Produces: + - `build_scheduler(cfg: AgentConfig, *, scheduler_class: type = BlockingScheduler) -> BaseScheduler` — registers job id `"heartbeat"` on an interval of `cfg.heartbeat_interval_minutes` minutes. `scheduler_class` exists for tests (BackgroundScheduler). + - `python -m agent` — startup sequence: load config → `ensure_database` → `run_migrations` → `run_heartbeat` once → scheduler runs forever. + - `python -m agent.healthcheck` — exit 0 when agentdb answers `SELECT 1`, exit 1 otherwise (Docker HEALTHCHECK). + +- [ ] **Step 1: Write the failing test** — `agent/tests/test_scheduler.py` + +```python +from datetime import timedelta + +from apscheduler.schedulers.background import BackgroundScheduler + +from agent.config import AgentConfig +from agent.scheduler import build_scheduler + + +def _cfg(minutes: int) -> AgentConfig: + return AgentConfig( + db_host="db", + db_port=5432, + db_user="u", + db_password="p", + agent_db_name="agentdb", + firefly_url="http://app:8080", + firefly_token="tok", + heartbeat_interval_minutes=minutes, + ) + + +def test_build_scheduler_registers_heartbeat_at_configured_interval(): + scheduler = build_scheduler(_cfg(minutes=15), scheduler_class=BackgroundScheduler) + scheduler.start(paused=True) + try: + job = scheduler.get_job("heartbeat") + assert job is not None + assert job.trigger.interval == timedelta(minutes=15) + finally: + scheduler.shutdown(wait=False) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd agent && pytest tests/test_scheduler.py -v` — Expected: FAIL with `ModuleNotFoundError: No module named 'agent.scheduler'` + +- [ ] **Step 3: Implement** — `agent/src/agent/scheduler.py` + +```python +"""APScheduler assembly: one place where jobs get registered.""" + +from __future__ import annotations + +from apscheduler.schedulers.base import BaseScheduler +from apscheduler.schedulers.blocking import BlockingScheduler + +from agent.config import AgentConfig +from agent.jobs.heartbeat import run_heartbeat + + +def build_scheduler( + cfg: AgentConfig, *, scheduler_class: type = BlockingScheduler +) -> BaseScheduler: + scheduler = scheduler_class(timezone="UTC") + scheduler.add_job( + run_heartbeat, + "interval", + args=[cfg], + minutes=cfg.heartbeat_interval_minutes, + id="heartbeat", + ) + return scheduler +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd agent && pytest tests/test_scheduler.py -v` — Expected: PASS + +- [ ] **Step 5: Implement the healthcheck** — `agent/src/agent/healthcheck.py` + +Glue over tested parts; no dedicated test. + +```python +"""Docker HEALTHCHECK entry: healthy = agentdb answers SELECT 1.""" + +from __future__ import annotations + +import sys + +from agent.config import load_config +from agent.db import connect + + +def main() -> int: + try: + cfg = load_config() + with connect(cfg) as conn: + conn.execute("SELECT 1") + return 0 + except Exception as exc: + print(f"unhealthy: {exc}", file=sys.stderr) + return 1 + + +if __name__ == "__main__": + sys.exit(main()) +``` + +- [ ] **Step 6: Implement the entrypoint** — `agent/src/agent/__main__.py` + +```python +"""Agent entrypoint: bootstrap the database, then run scheduled jobs forever.""" + +from __future__ import annotations + +import logging + +from agent import __version__ +from agent.config import load_config +from agent.db import ensure_database, run_migrations +from agent.jobs.heartbeat import run_heartbeat +from agent.scheduler import build_scheduler + +logging.basicConfig( + level=logging.INFO, + format="%(asctime)s %(levelname)s %(name)s: %(message)s", +) +logger = logging.getLogger("agent") + + +def main() -> None: + logger.info("firefly-agent %s starting", __version__) + cfg = load_config() + ensure_database(cfg) + applied = run_migrations(cfg) + if applied: + logger.info("applied migrations: %s", ", ".join(applied)) + status = run_heartbeat(cfg) + logger.info("startup heartbeat: %s", status) + logger.info( + "scheduler starting; heartbeat every %d minutes", + cfg.heartbeat_interval_minutes, + ) + build_scheduler(cfg).start() + + +if __name__ == "__main__": + main() +``` + +- [ ] **Step 7: Run the full suite** + +Run: `cd agent && pytest -v` — Expected: all tests PASS + +- [ ] **Step 8: Commit** + +```bash +git add agent/src/agent/scheduler.py agent/src/agent/healthcheck.py agent/src/agent/__main__.py agent/tests/test_scheduler.py +git commit -m "feat(agent): scheduler, healthcheck, and container entrypoint" +``` + +--- + +### Task 7: Verify CLI + +**Files:** +- Create: `agent/src/agent/verify.py` +- Test: `agent/tests/test_verify.py` + +**Interfaces:** +- Consumes: everything above. +- Produces: `python -m agent.verify` — post-deploy check printing: migrations applied + last sync runs (from agentdb), Firefly version, and the list of asset account names. Internal function `verify(cfg: AgentConfig, client: FireflyClient | None = None) -> list[str]` returns the report lines (printing wrapper around it); `client` param for tests. + +- [ ] **Step 1: Write the failing test** — `agent/tests/test_verify.py` + +```python +from agent.db import ensure_database, run_migrations +from agent.jobs.heartbeat import run_heartbeat +from agent.verify import verify + + +class FakeFirefly: + def about(self) -> dict: + return {"version": "6.1.1"} + + def accounts(self, type: str = "asset") -> list[dict]: + return [ + {"id": "1", "attributes": {"name": "NFCU Checking"}}, + {"id": "2", "attributes": {"name": "Venmo"}}, + ] + + +def test_verify_reports_db_firefly_and_accounts(cfg): + ensure_database(cfg) + run_migrations(cfg) + run_heartbeat(cfg, client=FakeFirefly()) + + report = "\n".join(verify(cfg, client=FakeFirefly())) + + assert "1 migration(s) applied" in report + assert "heartbeat ok" in report + assert "firefly: OK (version 6.1.1)" in report + assert "2 asset account(s)" in report + assert "NFCU Checking" in report +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd agent && pytest tests/test_verify.py -v` — Expected: FAIL with `ModuleNotFoundError: No module named 'agent.verify'` + +- [ ] **Step 3: Implement** — `agent/src/agent/verify.py` + +```python +"""Post-deploy verification: `python -m agent.verify`.""" + +from __future__ import annotations + +from agent.config import AgentConfig, load_config +from agent.db import connect +from agent.firefly import FireflyClient + + +def verify(cfg: AgentConfig, client: FireflyClient | None = None) -> list[str]: + lines: list[str] = [] + + with connect(cfg) as conn: + migrations = conn.execute("SELECT count(*) FROM schema_migrations").fetchone()[0] + runs = conn.execute( + "SELECT connector, status, finished_at" + " FROM sync_runs ORDER BY id DESC LIMIT 3" + ).fetchall() + lines.append(f"agentdb: OK ({migrations} migration(s) applied)") + for connector, status, finished_at in runs: + lines.append(f" last run: {connector} {status} at {finished_at}") + + if client is None: + client = FireflyClient(cfg.firefly_url, cfg.firefly_token) + info = client.about() + lines.append(f"firefly: OK (version {info.get('version', 'unknown')})") + + accounts = client.accounts() + lines.append(f"firefly: {len(accounts)} asset account(s):") + for account in accounts: + lines.append(f" - {account['attributes']['name']}") + return lines + + +def main() -> int: + for line in verify(load_config()): + print(line) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd agent && pytest tests/test_verify.py -v` — Expected: PASS + +- [ ] **Step 5: Commit** + +```bash +git add agent/src/agent/verify.py agent/tests/test_verify.py +git commit -m "feat(agent): post-deploy verify command" +``` + +--- + +### Task 8: Dockerfile, compose service, env docs, runbook + +**Files:** +- Create: `agent/Dockerfile`, `agent/.dockerignore`, `agent/README.md` +- Modify: `docker-compose.yml` (add `agent` service after `importer`), `.env.example` (new section), `README.md` (repo root, one line) + +**Interfaces:** +- Consumes: `python -m agent` and `python -m agent.healthcheck` from Task 6. +- Produces: `docker compose build agent` succeeds; `docker compose config -q` validates; `.env.example` documents the four new variables. + +- [ ] **Step 1: Create `agent/.dockerignore`** + +``` +tests/ +__pycache__/ +*.egg-info/ +.pytest_cache/ +``` + +- [ ] **Step 2: Create `agent/Dockerfile`** + +```dockerfile +FROM python:3.12-slim + +WORKDIR /app + +COPY pyproject.toml ./ +COPY src ./src + +RUN pip install --no-cache-dir . + +HEALTHCHECK --interval=60s --timeout=10s --start-period=30s \ + CMD ["python", "-m", "agent.healthcheck"] + +CMD ["python", "-m", "agent"] +``` + +- [ ] **Step 3: Build the image to verify** + +Run: `docker build -t firefly-agent ./agent` — Expected: builds successfully. + +- [ ] **Step 4: Add the `agent` service to `docker-compose.yml`** + +Insert after the `importer` service, matching the file's existing style: + +```yaml + agent: + build: ./agent + restart: unless-stopped + env_file: .env + networks: + - firefly + depends_on: + db: + condition: service_healthy + app: + condition: service_healthy +``` + +Note: no Traefik labels and no `traefik_proxy` network — the agent is internal-only (Global Constraints). + +- [ ] **Step 5: Add the agent section to `.env.example`** + +Append: + +``` +# --- Accounting Agent --- +# Firefly API base URL as seen from inside the compose network +FIREFLY_API_URL=http://app:8080 +# Personal Access Token: Firefly UI -> Options -> Profile -> OAuth -> Personal Access Tokens +AGENT_FIREFLY_TOKEN=REPLACE_WITH_PERSONAL_ACCESS_TOKEN +# Name of the agent's own database inside the existing Postgres container +AGENT_DB_NAME=agentdb +HEARTBEAT_INTERVAL_MINUTES=60 +``` + +- [ ] **Step 6: Validate compose config** + +Run: `docker compose config -q` — Expected: exit 0, no output. (Requires the local `.env` to exist; placeholder values are fine.) + +- [ ] **Step 7: Write `agent/README.md`** + +```markdown +# firefly-agent + +Worker container for the personal accounting agent. See the architecture spec: +`docs/superpowers/specs/2026-08-23-accounting-agent-architecture-design.md`. + +## What it does (Phase 1) + +On startup: creates `agentdb` in the stack's Postgres if missing, applies SQL +migrations from `src/agent/migrations/`, runs one heartbeat, then schedules the +heartbeat every `HEARTBEAT_INTERVAL_MINUTES`. The heartbeat calls Firefly's +`/api/v1/about` and records the outcome in `sync_runs`. + +## Development + + python3 -m venv .venv && source .venv/bin/activate + pip install -e './agent[dev]' + cd agent && pytest # needs Docker running (testcontainers) + +## Configuration + +All via environment variables (see `.env.example`, "Accounting Agent" section): +`FIREFLY_API_URL`, `AGENT_FIREFLY_TOKEN` (required), `AGENT_DB_NAME`, +`HEARTBEAT_INTERVAL_MINUTES`, plus the existing `DB_HOST`/`DB_PORT`/ +`POSTGRES_USER`/`POSTGRES_PASSWORD`. + +## Deploy & verify + + # on the host, in the repo directory + git pull + docker compose build agent + docker compose up -d agent + docker compose exec agent python -m agent.verify + +Expected verify output: agentdb OK with migrations applied, recent heartbeat +runs, Firefly version, and your asset accounts listed. + +Inspect heartbeats directly: + + docker compose exec db psql -U firefly -d agentdb \ + -c "SELECT connector, status, finished_at FROM sync_runs ORDER BY id DESC LIMIT 5;" +``` + +- [ ] **Step 8: Add one line to the repo-root `README.md`** + +```markdown +# Firefly III - jbnel.dev Finance Stack + +`agent/` — personal accounting agent worker; see `agent/README.md` and `docs/superpowers/specs/`. +``` + +- [ ] **Step 9: Run the full test suite one more time** + +Run: `cd agent && pytest -v` — Expected: all PASS + +- [ ] **Step 10: Commit** + +```bash +git add agent/Dockerfile agent/.dockerignore agent/README.md docker-compose.yml .env.example README.md +git commit -m "feat(agent): containerize agent and wire into compose stack" +``` + +--- + +### Task 9: Production setup and deploy (human-in-the-loop with Jake) + +No new code. This task configures Firefly and deploys the agent. Steps marked **[Jake]** need his hands (credentials, UI clicks); the rest can be driven together over SSH once the workflow is confirmed. + +- [ ] **Step 1: [Jake] Confirm the deploy workflow** + +The plan assumes: SSH to the host running the stack, `git pull` in the repo directory, `docker compose` commands there. Confirm or correct; record the actual workflow in `agent/README.md` if it differs. + +- [ ] **Step 2: [Jake] Create the Personal Access Token** + +Firefly UI (`https://firefly.jbnel.dev`) → **Options → Profile → OAuth → Personal Access Tokens → Create new token**, name it `agent`. Copy the token (shown once). + +- [ ] **Step 3: [Jake] Create asset accounts in Firefly** + +Firefly UI → **Accounts → Asset accounts → Create**. One per real account; suggested starting set (adjust to reality): NFCU Checking, NFCU Savings, TFCU Checking, Cash App, Venmo, PayPal. Opening balances optional — Phase 2 ingestion will populate transactions. + +- [ ] **Step 4: [Jake] Add agent variables to the host `.env`** + +Append the "Accounting Agent" section from `.env.example` to the host's `.env`, with the real token in `AGENT_FIREFLY_TOKEN`. + +- [ ] **Step 5: Deploy** + +On the host, in the repo directory: + +```bash +git pull +docker compose build agent +docker compose up -d agent +docker compose ps agent # wait for status: healthy +``` + +- [ ] **Step 6: Verify** + +```bash +docker compose exec agent python -m agent.verify +``` + +Expected: `agentdb: OK (1 migration(s) applied)`, a `heartbeat ok` run, `firefly: OK (version 6.x)`, and the asset accounts from Step 3 listed. If `firefly:` shows 401, the token is wrong; if connection refused, check `FIREFLY_API_URL=http://app:8080`. + +- [ ] **Step 7: Confirm the schedule is live** + +After one heartbeat interval (default 60 min — or set `HEARTBEAT_INTERVAL_MINUTES=2` temporarily and `docker compose up -d agent` to recreate): + +```bash +docker compose exec db psql -U firefly -d agentdb \ + -c "SELECT connector, status, finished_at FROM sync_runs ORDER BY id DESC LIMIT 5;" +``` + +Expected: multiple `heartbeat | ok` rows with advancing timestamps. Restore the interval if it was lowered. + +- [ ] **Step 8: Close out Phase 1** + +Mark this plan's checkboxes done and commit any runbook corrections discovered during deploy: + +```bash +git add -A && git commit -m "docs(agent): record actual deploy workflow from first production deploy" +``` + +Phase 1's definition of done: the agent container runs healthy on the host, heartbeats accumulate in `sync_runs`, and `python -m agent.verify` passes against production Firefly with real asset accounts listed.