Add personal accounting agent architecture design spec
Master architecture and phase decomposition for the agent system built around the Firefly III stack: connector-based ingestion (SimpleFIN primary), hybrid worker + Claude runtime, prep-and-approve bill pay, three-channel notifications, and a six-phase build order. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uv7wnzF6Xv6fmaRjFTiMMq
This commit is contained in:
Claude Fable 5
@@ -0,0 +1,151 @@
|
||||
# Personal Accounting Agent — Architecture Design
|
||||
|
||||
**Date:** 2026-08-23
|
||||
**Status:** Approved (Jake, 2026-08-23)
|
||||
**Scope:** Master architecture and phase decomposition. Each phase gets its own spec and implementation plan; this document is the frame they hang on.
|
||||
|
||||
## Purpose
|
||||
|
||||
Build a personal accounting agent around the existing Firefly III stack at
|
||||
`https://firefly.jbnel.dev`. The agent:
|
||||
|
||||
1. Monitors account balances across all institutions.
|
||||
2. Assesses spending trends, tracks budgets, and forecasts cashflow.
|
||||
3. Prepares utility and other bill payments, executing only with Jake's explicit approval.
|
||||
4. Alerts Jake to pertinent information through Telegram, ntfy, and email.
|
||||
5. Maintains account records (statements, bill PDFs) in a structured local folder, with cloud sync later.
|
||||
6. Keeps all record data queryable — ledger data in Firefly, everything else in a dedicated agent database.
|
||||
|
||||
## Decisions
|
||||
|
||||
These were settled in the design conversation and bind all phases:
|
||||
|
||||
| Decision | Choice |
|
||||
|---|---|
|
||||
| Primary bank feed | SimpleFIN Bridge (read-only, ~$1.50/mo) |
|
||||
| Coverage strategy | Connector layer: SimpleFIN plus per-institution connectors (CSV drop folder, email parsing, browser automation) until NFCU, TFCU, Cash App, Venmo, and PayPal are all covered. Verify real coverage per institution early; assume nothing. |
|
||||
| Bill-pay autonomy | Prep + approval. The agent tracks, verifies funding, and asks. Money never moves without an explicit approval from Jake. |
|
||||
| Runtime | Hybrid. One Python worker container does deterministic work on schedules; scheduled Claude sessions do judgment work (analysis, forecast narrative, anomaly triage). |
|
||||
| Notification channels | Telegram bot (interactive: alerts, approvals), ntfy (urgent push), email (digests, receipts). Routed by message class. |
|
||||
| Document acquisition | Dedicated finance email inbox via IMAP first; browser-automation downloads later for billers that never email PDFs. |
|
||||
| Business accounts | Separate Firefly instance when a venture opens. This instance stays purely personal. The connector layer is account-agnostic so it can serve both. |
|
||||
| Agent state | `agentdb`: a separate database in the existing Postgres container. |
|
||||
|
||||
## Architecture
|
||||
|
||||
One new container, `agent`, joins the existing compose stack (Firefly III core,
|
||||
Data Importer, Postgres 16, Redis, cron) behind Traefik. The worker is a Python
|
||||
3.12 application under `agent/` in this repo, structured as modules with hard
|
||||
boundaries so any module can become its own service during the planned k8s
|
||||
migration:
|
||||
|
||||
- **Connectors** — pull data from sources on schedules. Each connector
|
||||
implements one interface: fetch → emit canonical records. Initial set:
|
||||
SimpleFIN poller, CSV drop-folder watcher, IMAP inbox poller. Later: browser
|
||||
automation jobs.
|
||||
- **Sync engine** — normalizes connector output into canonical models
|
||||
(transaction, balance, document, bill), deduplicates, and pushes to Firefly
|
||||
via its REST API using a personal access token.
|
||||
- **Notifier** — one interface, three sinks (Telegram, ntfy, SMTP). Message
|
||||
class determines routing: interactive requests and real-time alerts go to
|
||||
Telegram, urgent one-way pushes to ntfy, digests and receipts to email.
|
||||
- **Rules engine** — deterministic checks after each sync: low balance, large
|
||||
or unusual transaction, failed sync, upcoming bill without positioned funds.
|
||||
- **Bill module** — maintains the bill calendar in `agentdb` from Firefly
|
||||
bills and parsed bill emails; runs the approval workflow.
|
||||
- **Scheduler** — APScheduler; every job idempotent and individually
|
||||
triggerable for testing.
|
||||
|
||||
Claude sessions run on schedules outside the container (Claude Code
|
||||
routines/cron). They read the Firefly API and `agentdb`, write analysis and
|
||||
narratives out through the notifier, and never write directly to the ledger.
|
||||
The agent container exposes the notifier to them as one token-authenticated
|
||||
HTTP endpoint (`POST /notify`), reachable only on the internal Docker network
|
||||
or via SSH tunnel — never through Traefik.
|
||||
|
||||
### Data flow
|
||||
|
||||
```
|
||||
SimpleFIN ─┐
|
||||
CSV drop ─┼→ Connectors → canonical models → dedup → Firefly API (ledger)
|
||||
IMAP ─┤ │
|
||||
Browser ─┘ └→ agentdb (documents, bills,
|
||||
sync runs, approvals)
|
||||
Firefly API + agentdb → Claude sessions → Notifier → Telegram / ntfy / email
|
||||
```
|
||||
|
||||
Deduplication: use the source's external ID where one exists (SimpleFIN
|
||||
provides one); otherwise a content hash of date + amount + account + normalized
|
||||
description. Firefly's own duplicate detection is the backstop, never the
|
||||
primary mechanism.
|
||||
|
||||
### agentdb schema (v1)
|
||||
|
||||
- `sync_runs` — one row per connector execution: connector, started, finished, status, counts, error.
|
||||
- `documents` — index of filed records: path, sha256, institution, account, doc type, period, source, received date.
|
||||
- `bills` — calendar: payee, expected amount, due date, funding account, autopay flag, status.
|
||||
- `approvals` — bill-pay approvals: bill, amount, requested at, approved/denied at, Telegram message ref, executed at.
|
||||
|
||||
## Safety and error handling
|
||||
|
||||
- **Idempotent imports.** Dedup keys make every retry safe.
|
||||
- **Escalating failure alerts.** Every connector run logs to `sync_runs`; three
|
||||
consecutive failures for a connector escalate to ntfy.
|
||||
- **Secrets stay server-side.** `.env` on the host only, never in the repo.
|
||||
Aggregator access is read-only.
|
||||
- **Approval integrity.** An approval exists only as an `approvals` row created
|
||||
by Jake's own Telegram interaction. The executor re-checks the approval
|
||||
immediately before acting and refuses stale or amount-mismatched approvals.
|
||||
- **Payment receipts.** Every browser-automation payment run captures
|
||||
screenshots and files them as documents.
|
||||
|
||||
## Documents
|
||||
|
||||
Local folder structure (exact layout specified in the Phase 5 spec):
|
||||
`records/<institution>/<account>/<year>/` with normalized filenames
|
||||
(`2026-08-nfcu-checking-statement.pdf`). Every filed document gets a
|
||||
`documents` row. Cloud sync via rclone; Jake picks the target during Phase 5.
|
||||
|
||||
## Testing
|
||||
|
||||
TDD throughout (superpowers test-driven-development skill). Connectors test
|
||||
against recorded fixtures: captured SimpleFIN JSON, real Venmo/Cash App/PayPal
|
||||
CSV exports, sample bill emails. The Firefly client tests against a throwaway
|
||||
local Firefly via a compose test profile. One end-to-end smoke test proves
|
||||
fixture → dedup → Firefly.
|
||||
|
||||
## Deployment
|
||||
|
||||
The stack runs on a remote host behind Traefik; this repo is the config source.
|
||||
Assumed workflow: SSH to the host, `git pull`, `docker compose up -d`. Confirm
|
||||
the exact workflow during Phase 1 planning and record it in the Phase 1 spec.
|
||||
|
||||
## Phases
|
||||
|
||||
Each phase is its own sub-project: spec → plan → implementation. Later phases
|
||||
may not start until the prior phase runs in production.
|
||||
|
||||
1. **Foundation** — Firefly configured for API use (personal access token,
|
||||
asset accounts created); `agent/` skeleton: Dockerfile, config loading,
|
||||
scheduler with a heartbeat job, `agentdb` migrations, compose service,
|
||||
deployed to the host.
|
||||
2. **Ingestion** — SimpleFIN → Firefly sync with dedup; verified coverage for
|
||||
NFCU and TFCU; CSV drop-folder connector with mappers for Venmo, Cash App,
|
||||
and PayPal exports.
|
||||
3. **Alerts & monitoring** — notifier with all three sinks; rules engine:
|
||||
low balance, large/unusual transaction, sync failure, negative trend.
|
||||
4. **Analysis & forecasting** — scheduled Claude sessions: weekly spending
|
||||
review, budget tracking, recurring-transaction detection, 30/60/90-day
|
||||
cashflow forecast, monthly digest.
|
||||
5. **Documents** — finance inbox via IMAP, PDF extraction and filing, document
|
||||
index, cloud sync target chosen and wired.
|
||||
6. **Bills** — bill calendar, funding checks, Telegram approve/deny flow,
|
||||
browser automation for approved payments and statement downloads.
|
||||
7. **Later** — business Firefly instance when a venture opens; k8s migration
|
||||
per `k8s/README.md`.
|
||||
|
||||
## Out of scope
|
||||
|
||||
- Investment portfolio management and tax preparation.
|
||||
- Multi-user support; this system serves Jake alone.
|
||||
- Modifying Firefly III itself; the agent is strictly an API consumer.
|
||||
Reference in New Issue
Block a user