07 — Monitoring, Logging & Admin Dashboard
Ring: 1 (basic logging) + Ring 2 (dashboard UI + user dashboard) Dependency: R1-1 (Auth), R1-2 (Cost Control — usage_daily table) Handbook: Ch. 47 (observability), Ch. 206-210 (monitoring, security) Related: Hetzner migration plan — Uptime Kuma (infra monitoring)
Problem
- We don’t know what users are doing — no activity log exists.
- Admin panel only contains segment CRUD.
- AI cost is invisible (addressed by 02-api-cost-control but no visualization).
- Pipeline health is unmonitored (latency, error rate, accuracy).
- No user dashboard — after login, goes directly to the companies page.
- No click-level analytics (which button, which flow, where users drop off).
Decisions
D1: 3-Layer Monitoring
D2: Activity Log — What Gets Logged
Ring 1 (launch):| Action Category | Examples |
|---|---|
| Discovery | discovery.started, discovery.completed, discovery.failed |
| Headhunt | headhunt.started, headhunt.completed |
| Lead | lead.created, lead.updated, lead.deleted |
| Interaction | interaction.created |
| Task | task.created, task.completed |
| Batch | batch.started, batch.completed, batch.failed |
| Auth | user.login, user.logout, user.signup |
| Org | org.created, member.invited, member.removed, settings.changed |
| Billing | plan.upgraded, plan.downgraded, credit.purchased, credit.consumed |
| View | company.viewed, lead.viewed, contact.viewed |
| Action | Tool |
|---|---|
| Button clicks | PostHog |
| Filter/sort changes | PostHog |
| Scroll depth | PostHog |
| Session duration | PostHog |
| Funnel analysis (signup → first discovery → first lead) | PostHog |
| Heatmaps | PostHog |
| Session replay | PostHog |
DB activity_log = BUSINESS actions (queryable, integrated with billing). PostHog = UX actions (visual analysis, product decisions).
D3: Dashboard Pages
Admin Dashboard (super_admin —/admin/*):
| Page | Content | Ring |
|---|---|---|
/admin/dashboard | Overview: today’s cost, active users, error count, pipeline health score, recent actions | R2 |
/admin/usage | Cost detail: daily/weekly/monthly chart, by provider/route/org | R2 |
/admin/users | User/org list: plan, last activity, usage rate | R2 |
/admin/pipeline | Pipeline health: average latency, error rate, accuracy, per stage | R2 |
/admin/activity | Activity log: filterable table (user, action, date, org) | R2 |
/admin/segments | Existing: Segment CRUD | ✅ EXISTS |
/admin/providers | AI provider comparison: cost, latency, error rate | R2 |
/admin/search-trends | Popular keywords, countries, industries | R2 |
/dashboard) — Ring 2:
D4: User Dashboard Timing
- Ring 1: None. After login →
/discoverypage (same as current). - Ring 2:
/dashboardpage is created, becomes the landing page after login.
Data Model
New Table: Activity Log
New Table: API Metrics (optional — Ring 2)
This table logs every API request. Implemented in Ring 2. In Ring 1, activity_log alone is sufficient.
Architecture
Activity Logger
logActivity() is async + fire-and-forget (await is optional). Logging errors DO NOT BLOCK user operations.
Middleware-Based API Metrics (Ring 2)
PostHog Integration (Ring 2-3)
Current Code Impact
New Files (Ring 1)
| File | Content |
|---|---|
lib/logging/activity.ts | logActivity() + IActivityEvent |
lib/logging/types.ts | Action type enum/const |
New Files (Ring 2)
| File | Content |
|---|---|
app/admin/dashboard/page.tsx | Admin overview |
app/admin/usage/page.tsx | Cost detail + charts |
app/admin/users/page.tsx | User/org management |
app/admin/pipeline/page.tsx | Pipeline health |
app/admin/activity/page.tsx | Activity log (filterable) |
app/admin/providers/page.tsx | AI provider comparison |
app/admin/search-trends/page.tsx | Search trends |
app/dashboard/page.tsx | User dashboard (landing) |
Files to Change (Ring 1)
| File | Change |
|---|---|
| All API routes | Add logActivity() call |
middleware.ts | API metrics logging (Ring 2) |
Data Retention
| Table | Retention Period | Reason |
|---|---|---|
activity_log | 90 days (then archive/delete) | Generates a lot of data, old records unnecessary |
api_metrics | 30 days (then aggregate + delete) | Raw metrics only for short-term debugging |
usage_daily | Unlimited | Aggregated data, small, needed for billing |
ai_job_runs | 180 days | Cost analysis + debugging |
| PostHog | 90 days (self-hosted setting) | Session replay takes a lot of space |
Batch job (cron) to clean old data will be added in Ring 2.
Future Decisions
FD-1: PostHog Self-Hosted (Ring 2-3)
Deploy via Coolify on Hetzner. Click tracking, heatmaps, session replay, funnel analysis. Cost: $0. Data stays local.
FD-2: Anomaly Detection (Ring 3+)
Learn normal usage patterns, alert on deviations. “Today’s cost is 10x yesterday’s” → toast + email.
FD-3: Custom Admin Dashboards (Ring 3)
Admin can select and arrange their own widgets. Drag-and-drop layout.
FD-4: Data Warehouse / Export (Ring 4)
Activity log + metrics → BigQuery or ClickHouse export. For advanced analysis.
FD-5: Real-time Dashboard (Ring 4)
Live-updating admin dashboard via Supabase Realtime or WebSocket.
Atomic Tasks
| # | Task | Ring | Size |
|---|---|---|---|
| MON-1 | export_ai_activity_log table + RLS + index | R1 | Migration |
| MON-2 | lib/logging/activity.ts — logActivity() + types | R1 | Small |
| MON-3 | logActivity() integration in all API routes | R1 | Medium |
| MON-4 | export_ai_api_metrics table + middleware logging | R2 | Medium |
| MON-5 | /admin/dashboard — overview | R2 | Large |
| MON-6 | /admin/usage — cost charts (Recharts or Chart.js) | R2 | Large |
| MON-7 | /admin/users — user/org list + detail | R2 | Medium |
| MON-8 | /admin/pipeline — latency, error rate, accuracy | R2 | Medium |
| MON-9 | /admin/activity — filterable log table | R2 | Medium |
| MON-10 | /admin/providers — AI provider comparison | R2 | Small |
| MON-11 | /admin/search-trends — popular searches | R2 | Small |
| MON-12 | /dashboard — user dashboard (landing page) | R2 | Large |
| MON-13 | PostHog self-hosted deploy (Hetzner) | R2-3 | Medium |
| MON-14 | PostHog client-side SDK integration | R2-3 | Small |
| MON-15 | Data retention cron job (old record cleanup) | R2 | Small |