Documentation Index
Fetch the complete documentation index at: https://cernio.gadulabs.com/llms.txt
Use this file to discover all available pages before exploring further.
07 — Monitoring, Logging & Admin Dashboard
Ring: 1 (basic logging) + Ring 2 (dashboard UI + user dashboard)
Dependency: R1-1 (Auth), R1-2 (Cost Control — usage_daily table)
Handbook: Ch. 47 (observability), Ch. 206-210 (monitoring, security)
Related: Hetzner migration plan — Uptime Kuma (infra monitoring)
Problem
- We don’t know what users are doing — no activity log exists.
- Admin panel only contains segment CRUD.
- AI cost is invisible (addressed by 02-api-cost-control but no visualization).
- Pipeline health is unmonitored (latency, error rate, accuracy).
- No user dashboard — after login, goes directly to the companies page.
- No click-level analytics (which button, which flow, where users drop off).
Decisions
D1: 3-Layer Monitoring
LAYER 1: Infrastructure (READY in Hetzner plan)
→ Uptime Kuma: site/API uptime, response time
→ Telegram alert: CPU/RAM threshold breach
→ Coolify: container health
→ NO ADDITIONAL WORK — defined in migration plan
LAYER 2: Application Monitoring (Ring 1-2)
→ Activity log: every significant action written to DB
→ AI cost tracking: usage_daily + ai_job_runs (with 02)
→ API metrics: response time, error rate per route
→ Pipeline metrics: latency per stage, accuracy
LAYER 3: Product Analytics — Click-Level (Ring 2-3)
→ PostHog self-hosted (Hetzner VPS, deployed via Coolify)
→ Heatmaps, session replay, funnel analysis
→ Data stays local, GDPR compliant
→ Cost: $0 (self-hosted)
D2: Activity Log — What Gets Logged
Ring 1 (launch):
| Action Category | Examples |
|---|
| Discovery | discovery.started, discovery.completed, discovery.failed |
| Headhunt | headhunt.started, headhunt.completed |
| Lead | lead.created, lead.updated, lead.deleted |
| Interaction | interaction.created |
| Task | task.created, task.completed |
| Batch | batch.started, batch.completed, batch.failed |
| Auth | user.login, user.logout, user.signup |
| Org | org.created, member.invited, member.removed, settings.changed |
| Billing | plan.upgraded, plan.downgraded, credit.purchased, credit.consumed |
| View | company.viewed, lead.viewed, contact.viewed |
Ring 2-3 (after PostHog is added):
| Action | Tool |
|---|
| Button clicks | PostHog |
| Filter/sort changes | PostHog |
| Scroll depth | PostHog |
| Session duration | PostHog |
| Funnel analysis (signup → first discovery → first lead) | PostHog |
| Heatmaps | PostHog |
| Session replay | PostHog |
DB activity_log = BUSINESS actions (queryable, integrated with billing).
PostHog = UX actions (visual analysis, product decisions).
D3: Dashboard Pages
Admin Dashboard (super_admin — /admin/*):
| Page | Content | Ring |
|---|
/admin/dashboard | Overview: today’s cost, active users, error count, pipeline health score, recent actions | R2 |
/admin/usage | Cost detail: daily/weekly/monthly chart, by provider/route/org | R2 |
/admin/users | User/org list: plan, last activity, usage rate | R2 |
/admin/pipeline | Pipeline health: average latency, error rate, accuracy, per stage | R2 |
/admin/activity | Activity log: filterable table (user, action, date, org) | R2 |
/admin/segments | Existing: Segment CRUD | ✅ EXISTS |
/admin/providers | AI provider comparison: cost, latency, error rate | R2 |
/admin/search-trends | Popular keywords, countries, industries | R2 |
User Dashboard (user — /dashboard) — Ring 2:
┌─────────────────────────────────────────────────────────────┐
│ Welcome, Alex [Pro Plan] │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌─── This Month ───────┐ ┌─── Quick Actions ───────────┐ │
│ │ 12 discoveries made │ │ [+ New Discovery] │ │
│ │ 47 companies found │ │ [Recent Leads] │ │
│ │ 5 leads created │ │ [Pending Tasks (3)] │ │
│ └──────────────────────┘ └──────────────────────────────┘ │
│ │
│ ┌─── Usage ────────────────────────────────────────────┐ │
│ │ Discovery: ████████████░░░░ 38/50 │ │
│ │ Contacts: ██████░░░░░░░░░░ 28/100 │ │
│ │ Credits: 23 remaining │ │
│ └──────────────────────────────────────────────────────┘ │
│ │
│ ┌─── Recent Searches ───────────────────┐ ┌── Leads ───┐ │
│ │ "chemicals" Germany — 12 co. (yesterday) │ Saved: 5 │ │
│ │ "textiles" Italy — 8 co. (3 days ago) │ Contact: 3 │ │
│ │ "machinery" Poland — 15 co. (1 week ago) │ Negot.: 1 │ │
│ └──────────────────────────────────────────┘ └──────────┘ │
│ │
│ ┌─── Upcoming Tasks ──────────────────────────────────┐ │
│ │ 🔴 TextilChem GmbH — Send catalog (overdue!) │ │
│ │ 🟡 ChemTrade AG — Follow up (tomorrow) │ │
│ │ 🟢 Mueller KG — Send price quote (3 days) │ │
│ └──────────────────────────────────────────────────────┘ │
│ │
│ ┌─── Team Activity (Team+) ───────────────────────────┐ │
│ │ Ahmet: discovered 3 companies (today) │ │
│ │ Mehmet: created 2 leads (yesterday) │ │
│ └──────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
D4: User Dashboard Timing
- Ring 1: None. After login →
/discovery page (same as current).
- Ring 2:
/dashboard page is created, becomes the landing page after login.
Data Model
New Table: Activity Log
CREATE TABLE export_ai_activity_log (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
organization_id UUID NOT NULL REFERENCES export_ai_organizations(id),
user_id UUID NOT NULL REFERENCES auth.users(id),
action TEXT NOT NULL, -- 'discovery.started', 'lead.created', etc.
resource_type TEXT, -- 'search', 'company', 'lead', 'contact', 'batch_job'
resource_id UUID, -- ID of the related record
metadata JSONB DEFAULT '{}', -- { keyword, country, cost, contacts_found, ... }
ip_address INET,
user_agent TEXT,
created_at TIMESTAMPTZ DEFAULT now()
);
-- RLS
ALTER TABLE export_ai_activity_log ENABLE ROW LEVEL SECURITY;
-- Indexes (for fast queries)
CREATE INDEX idx_activity_org_date ON export_ai_activity_log(organization_id, created_at DESC);
CREATE INDEX idx_activity_user ON export_ai_activity_log(user_id, created_at DESC);
CREATE INDEX idx_activity_action ON export_ai_activity_log(action, created_at DESC);
-- Old record cleanup (after 90 days — via batch job)
-- Or: partition by month (Ring 4 for large data)
New Table: API Metrics (optional — Ring 2)
CREATE TABLE export_ai_api_metrics (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
endpoint TEXT NOT NULL, -- '/api/discover', '/api/headhunt', ...
method TEXT NOT NULL, -- 'GET', 'POST'
status_code INT NOT NULL,
response_time_ms INT NOT NULL,
organization_id UUID,
user_id UUID,
created_at TIMESTAMPTZ DEFAULT now()
);
CREATE INDEX idx_api_metrics_endpoint ON export_ai_api_metrics(endpoint, created_at DESC);
This table logs every API request. Implemented in Ring 2. In Ring 1, activity_log alone is sufficient.
Architecture
Activity Logger
// lib/logging/activity.ts
interface IActivityEvent {
action: string; // 'discovery.started'
resourceType?: string; // 'search'
resourceId?: string;
metadata?: Record<string, unknown>;
}
async function logActivity(
session: ISessionContext,
event: IActivityEvent,
request?: Request
): Promise<void>
// Usage:
await logActivity(session, {
action: 'discovery.started',
resourceType: 'search',
metadata: { keyword: 'chemicals', country: 'DE' }
}, request);
logActivity() is async + fire-and-forget (await is optional). Logging errors DO NOT BLOCK user operations.
Middleware-Based API Metrics (Ring 2)
// Added to middleware.ts
// Logs every API request's response time + status code
// Async — does not delay the response
PostHog Integration (Ring 2-3)
On Hetzner VPS (Galata or Kadikoy):
→ Deploy PostHog self-hosted via Coolify
→ posthog.cernio.com (dashboard behind VPN)
→ Client-side: posthog-js SDK
→ Server-side: posthog-node SDK (optional)
Current Code Impact
New Files (Ring 1)
| File | Content |
|---|
lib/logging/activity.ts | logActivity() + IActivityEvent |
lib/logging/types.ts | Action type enum/const |
New Files (Ring 2)
| File | Content |
|---|
app/admin/dashboard/page.tsx | Admin overview |
app/admin/usage/page.tsx | Cost detail + charts |
app/admin/users/page.tsx | User/org management |
app/admin/pipeline/page.tsx | Pipeline health |
app/admin/activity/page.tsx | Activity log (filterable) |
app/admin/providers/page.tsx | AI provider comparison |
app/admin/search-trends/page.tsx | Search trends |
app/dashboard/page.tsx | User dashboard (landing) |
Files to Change (Ring 1)
| File | Change |
|---|
| All API routes | Add logActivity() call |
middleware.ts | API metrics logging (Ring 2) |
Data Retention
| Table | Retention Period | Reason |
|---|
activity_log | 90 days (then archive/delete) | Generates a lot of data, old records unnecessary |
api_metrics | 30 days (then aggregate + delete) | Raw metrics only for short-term debugging |
usage_daily | Unlimited | Aggregated data, small, needed for billing |
ai_job_runs | 180 days | Cost analysis + debugging |
| PostHog | 90 days (self-hosted setting) | Session replay takes a lot of space |
Batch job (cron) to clean old data will be added in Ring 2.
Future Decisions
FD-1: PostHog Self-Hosted (Ring 2-3)
Deploy via Coolify on Hetzner. Click tracking, heatmaps, session replay, funnel analysis. Cost: $0. Data stays local.
FD-2: Anomaly Detection (Ring 3+)
Learn normal usage patterns, alert on deviations. “Today’s cost is 10x yesterday’s” → toast + email.
FD-3: Custom Admin Dashboards (Ring 3)
Admin can select and arrange their own widgets. Drag-and-drop layout.
FD-4: Data Warehouse / Export (Ring 4)
Activity log + metrics → BigQuery or ClickHouse export. For advanced analysis.
FD-5: Real-time Dashboard (Ring 4)
Live-updating admin dashboard via Supabase Realtime or WebSocket.
Atomic Tasks
| # | Task | Ring | Size |
|---|
| MON-1 | export_ai_activity_log table + RLS + index | R1 | Migration |
| MON-2 | lib/logging/activity.ts — logActivity() + types | R1 | Small |
| MON-3 | logActivity() integration in all API routes | R1 | Medium |
| MON-4 | export_ai_api_metrics table + middleware logging | R2 | Medium |
| MON-5 | /admin/dashboard — overview | R2 | Large |
| MON-6 | /admin/usage — cost charts (Recharts or Chart.js) | R2 | Large |
| MON-7 | /admin/users — user/org list + detail | R2 | Medium |
| MON-8 | /admin/pipeline — latency, error rate, accuracy | R2 | Medium |
| MON-9 | /admin/activity — filterable log table | R2 | Medium |
| MON-10 | /admin/providers — AI provider comparison | R2 | Small |
| MON-11 | /admin/search-trends — popular searches | R2 | Small |
| MON-12 | /dashboard — user dashboard (landing page) | R2 | Large |
| MON-13 | PostHog self-hosted deploy (Hetzner) | R2-3 | Medium |
| MON-14 | PostHog client-side SDK integration | R2-3 | Small |
| MON-15 | Data retention cron job (old record cleanup) | R2 | Small |