restructure: flatten project, add ARCHITECTURE.md, update DEVY.md
- Moved from session-analytics-web/ subfolder to root - Renamed to token-analytics - Added deploy.sh, ARCHITECTURE.md - Moved export.mjs into jsonl-sync/
This commit is contained in:
1 parent
1d3070ed4c
commit
dbaad6b372
17 files changed
+340
-270
No files matched your search
+7
-18
@@ -1,21 +1,10 @@
|
|||||||
# Node
|
/.vs/
|
||||||
node_modules/
|
/bin/
|
||||||
|
/obj/
|
||||||
# .NET
|
/publish/
|
||||||
bin/
|
|
||||||
obj/
|
|
||||||
*.user
|
*.user
|
||||||
|
|
||||||
# Data
|
/jsonl-sync/node_modules/
|
||||||
sessions.db
|
/jsonl-sync/sessions.db
|
||||||
state.json
|
/jsonl-sync/state.json
|
||||||
|
|
||||||
# IDE
|
|
||||||
.vs/
|
|
||||||
.vscode/
|
|
||||||
*.swp
|
|
||||||
*.swo
|
|
||||||
|
|
||||||
# OS
|
|
||||||
.DS_Store
|
|
||||||
Thumbs.db
|
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
# Architecture
|
||||||
|
|
||||||
|
## Stack
|
||||||
|
|
||||||
|
- **Backend:** ASP.NET (.NET 10), Minimal API
|
||||||
|
- **Frontend:** Vanilla HTML/CSS/JS (SPA, kein Framework)
|
||||||
|
- **Datenbank:** SQLite via Dapper (read) + better-sqlite3 (import)
|
||||||
|
- **Port:** 18684
|
||||||
|
|
||||||
|
## Komponenten
|
||||||
|
|
||||||
|
### Backend (`Program.cs`)
|
||||||
|
Minimal API mit Endpunkten für Kosten-Analyse:
|
||||||
|
- `GET /api/cost-buckets` - Zeitraum in N Buckets aggregiert
|
||||||
|
- `GET /api/cost-entries` - Einzelne Einträge mit Kosten
|
||||||
|
- `GET /api/cost-chains` - Konversations-Ketten nach Kosten sortiert
|
||||||
|
- `GET /api/cost-summary` - Gesamtkosten für Zeitraum
|
||||||
|
- `GET /api/entry-chain/{id}` - Volle Kette eines Eintrags (parentId vor/zurück)
|
||||||
|
- `GET /api/entry-chains` - Einträge mit chainId-Zuordnung
|
||||||
|
- `POST /api/sync` - Triggert JSONL-Import via `export.mjs`
|
||||||
|
|
||||||
|
DB-Zugriff über Dapper + `Microsoft.Data.Sqlite`. Connection per Request via Factory.
|
||||||
|
|
||||||
|
### JSONL Sync (`jsonl-sync/export.mjs`)
|
||||||
|
Node.js Script das OpenClaw Session-JSONL-Files in SQLite importiert:
|
||||||
|
1. Liest alle `.jsonl` aus `/home/node/.openclaw/agents/main/sessions/`
|
||||||
|
2. Parsed Einträge, speichert in `entries` Tabelle
|
||||||
|
3. Berechnet `chainId` durch parentId-Walk bis zum nächsten User-Message
|
||||||
|
4. Inkrementell - merkt sich pro File die letzte verarbeitete Zeile
|
||||||
|
|
||||||
|
### Frontend (`wwwroot/`)
|
||||||
|
SPA mit Zoom-Navigation durch Zeiträume:
|
||||||
|
- **Bucket-View:** Balkendiagramm mit klickbarem Drill-Down (Monat → Woche → Tag → Stunden)
|
||||||
|
- **Entry-View:** Einzelne API-Calls als Balken, farblich nach Konversations-Kette
|
||||||
|
- **Chain-Blocks:** Teuerste Konversationen als klickbare Blöcke
|
||||||
|
- **Modal:** Konversations-Kontext mit User/Assistant Messages
|
||||||
|
- State wird im URL-Hash gespeichert
|
||||||
|
|
||||||
|
## Datenfluss
|
||||||
|
|
||||||
|
```
|
||||||
|
OpenClaw JSONL Files → export.mjs → SQLite (sessions.db) → .NET API → Frontend
|
||||||
|
```
|
||||||
|
|
||||||
|
## Deployment
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./deploy.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Baut Release, kopiert nach `publish/`, restartet den `token-analytics` systemd Service auf stein2.
|
||||||
@@ -1,13 +1,21 @@
|
|||||||
# JSONL Exporter
|
# Token Analytics
|
||||||
|
|
||||||
## Projektbeschreibung
|
## Projektbeschreibung
|
||||||
|
|
||||||
Exportiert OpenClaw Session-JSONL-Dateien als einzelne JSON-Files nach `workspace/logs/`. Inkrementell via `state.json`.
|
Cost-Analytics Dashboard für OpenClaw API-Nutzung. Importiert Session-JSONL-Files in SQLite, bietet eine REST API und ein interaktives Frontend mit Drill-Down von Monats- bis Einzeleintrags-Ebene. Konversations-Ketten werden über parentId-Walks rekonstruiert.
|
||||||
|
|
||||||
|
**Stack:** .NET 10 Minimal API + Vanilla JS SPA + SQLite | **Port:** 18684
|
||||||
|
|
||||||
## Pfad
|
## Pfad
|
||||||
|
|
||||||
`/home/node/.openclaw/workspace/projects/jsonl-exporter`
|
`/home/node/.openclaw/workspace/projects/token-analytics`
|
||||||
|
|
||||||
|
## Deployment
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./deploy.sh
|
||||||
|
```
|
||||||
|
|
||||||
## Notizen
|
## Notizen
|
||||||
|
|
||||||
-
|
- Siehe ARCHITECTURE.md für Details
|
||||||
@@ -1,29 +0,0 @@
|
|||||||
# JSONL Exporter
|
|
||||||
|
|
||||||
Liest alle Session-JSONL-Dateien aus dem OpenClaw Sessions-Verzeichnis und exportiert jeden Eintrag als einzelne JSON-Datei nach `workspace/logs/`.
|
|
||||||
|
|
||||||
## Funktionsweise
|
|
||||||
|
|
||||||
- Scannt `/home/node/.openclaw/agents/main/sessions/*.jsonl`
|
|
||||||
- Pro Eintrag wird eine Datei mit lesbarem Namen angelegt:
|
|
||||||
`YYYY-MM-DD HH-mm-ss ms - <shortId> (type).json`
|
|
||||||
- Speichert den Fortschritt in `state.json` (nur neue Einträge werden exportiert)
|
|
||||||
- Überspringt gelöschte Sessions (`.deleted`) und fehlerhafte Zeilen
|
|
||||||
|
|
||||||
## Dateien
|
|
||||||
|
|
||||||
| Datei | Zweck |
|
|
||||||
|-------|-------|
|
|
||||||
| `export.mjs` | Hauptskript (ESM) |
|
|
||||||
| `export.js` | Gleiche Logik als CommonJS |
|
|
||||||
| `state.json` | Fortschritts-State (verarbeitete Zeilenzahl pro JSONL) |
|
|
||||||
|
|
||||||
## Ausführen
|
|
||||||
|
|
||||||
```bash
|
|
||||||
node export.mjs
|
|
||||||
# oder
|
|
||||||
node export.js
|
|
||||||
```
|
|
||||||
|
|
||||||
Output landet in: `workspace/logs/`
|
|
||||||
@@ -3,8 +3,7 @@ using Microsoft.Data.Sqlite;
|
|||||||
|
|
||||||
var builder = WebApplication.CreateBuilder(args);
|
var builder = WebApplication.CreateBuilder(args);
|
||||||
|
|
||||||
var dbPath = builder.Configuration["DbPath"]
|
var dbPath = builder.Configuration["DbPath"]!;
|
||||||
?? Path.Combine(AppContext.BaseDirectory, "..", "..", "..", "..", "sessions.db");
|
|
||||||
|
|
||||||
builder.Services.AddSingleton<Func<SqliteConnection>>(() =>
|
builder.Services.AddSingleton<Func<SqliteConnection>>(() =>
|
||||||
new SqliteConnection($"Data Source={Path.GetFullPath(dbPath)}"));
|
new SqliteConnection($"Data Source={Path.GetFullPath(dbPath)}"));
|
||||||
@@ -37,7 +36,6 @@ app.MapGet("/api/cost-buckets", (Func<SqliteConnection> factory, string from, st
|
|||||||
});
|
});
|
||||||
|
|
||||||
// GET /api/cost-entries?from=&to=
|
// GET /api/cost-entries?from=&to=
|
||||||
// Returns all cost entries in a time range
|
|
||||||
app.MapGet("/api/cost-entries", (Func<SqliteConnection> factory, string from, string to) =>
|
app.MapGet("/api/cost-entries", (Func<SqliteConnection> factory, string from, string to) =>
|
||||||
{
|
{
|
||||||
using var db = factory();
|
using var db = factory();
|
||||||
@@ -47,6 +45,7 @@ app.MapGet("/api/cost-entries", (Func<SqliteConnection> factory, string from, st
|
|||||||
json_extract(data, '$.message.usage.input') AS inputTokens,
|
json_extract(data, '$.message.usage.input') AS inputTokens,
|
||||||
json_extract(data, '$.message.usage.output') AS outputTokens,
|
json_extract(data, '$.message.usage.output') AS outputTokens,
|
||||||
json_extract(data, '$.message.model') AS model,
|
json_extract(data, '$.message.model') AS model,
|
||||||
|
chainId,
|
||||||
data
|
data
|
||||||
FROM entries
|
FROM entries
|
||||||
WHERE json_extract(data, '$.message.usage.cost.total') IS NOT NULL
|
WHERE json_extract(data, '$.message.usage.cost.total') IS NOT NULL
|
||||||
@@ -57,6 +56,29 @@ app.MapGet("/api/cost-entries", (Func<SqliteConnection> factory, string from, st
|
|||||||
return Results.Ok(rows);
|
return Results.Ok(rows);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// GET /api/cost-chains?from=&to=
|
||||||
|
// Returns chains with aggregated cost, sorted by total cost desc
|
||||||
|
app.MapGet("/api/cost-chains", (Func<SqliteConnection> factory, string from, string to, int limit = 50, int offset = 0) =>
|
||||||
|
{
|
||||||
|
using var db = factory();
|
||||||
|
var rows = db.Query(@"
|
||||||
|
SELECT chainId,
|
||||||
|
SUM(CAST(json_extract(data, '$.message.usage.cost.total') AS REAL)) AS totalCost,
|
||||||
|
COUNT(*) AS count,
|
||||||
|
MIN(timestamp) AS first,
|
||||||
|
MAX(timestamp) AS last
|
||||||
|
FROM entries
|
||||||
|
WHERE json_extract(data, '$.message.usage.cost.total') IS NOT NULL
|
||||||
|
AND timestamp >= @from AND timestamp < @to
|
||||||
|
AND chainId IS NOT NULL
|
||||||
|
GROUP BY chainId
|
||||||
|
ORDER BY totalCost DESC
|
||||||
|
LIMIT @limit OFFSET @offset",
|
||||||
|
new { from, to, limit, offset });
|
||||||
|
|
||||||
|
return Results.Ok(rows);
|
||||||
|
});
|
||||||
|
|
||||||
// GET /api/cost-summary?from=&to=
|
// GET /api/cost-summary?from=&to=
|
||||||
app.MapGet("/api/cost-summary", (Func<SqliteConnection> factory, string? from, string? to) =>
|
app.MapGet("/api/cost-summary", (Func<SqliteConnection> factory, string? from, string? to) =>
|
||||||
{
|
{
|
||||||
@@ -151,81 +173,37 @@ app.MapGet("/api/entry-chain/{id}", (string id, Func<SqliteConnection> factory)
|
|||||||
return Results.Ok(fullChain);
|
return Results.Ok(fullChain);
|
||||||
});
|
});
|
||||||
|
|
||||||
// GET /api/entry-chains?from=&to= - returns cost entries with resolved chainId
|
// GET /api/entry-chains?from=&to= - returns cost entries with cached chainId
|
||||||
app.MapGet("/api/entry-chains", (Func<SqliteConnection> factory, string from, string to) =>
|
app.MapGet("/api/entry-chains", (Func<SqliteConnection> factory, string from, string to) =>
|
||||||
{
|
{
|
||||||
using var db = factory();
|
using var db = factory();
|
||||||
|
var rows = db.Query(@"
|
||||||
// Get all cost entries in range
|
|
||||||
var costEntries = db.Query(@"
|
|
||||||
SELECT id, session, type, timestamp,
|
SELECT id, session, type, timestamp,
|
||||||
json_extract(data, '$.message.usage.cost.total') AS cost,
|
json_extract(data, '$.message.usage.cost.total') AS cost,
|
||||||
json_extract(data, '$.message.usage.input') AS inputTokens,
|
json_extract(data, '$.message.usage.input') AS inputTokens,
|
||||||
json_extract(data, '$.message.usage.output') AS outputTokens,
|
json_extract(data, '$.message.usage.output') AS outputTokens,
|
||||||
json_extract(data, '$.message.model') AS model,
|
json_extract(data, '$.message.model') AS model,
|
||||||
json_extract(data, '$.parentId') AS parentId,
|
chainId,
|
||||||
data
|
data
|
||||||
FROM entries
|
FROM entries
|
||||||
WHERE json_extract(data, '$.message.usage.cost.total') IS NOT NULL
|
WHERE json_extract(data, '$.message.usage.cost.total') IS NOT NULL
|
||||||
AND timestamp >= @from AND timestamp < @to
|
AND timestamp >= @from AND timestamp < @to
|
||||||
ORDER BY timestamp ASC", new { from, to }).AsList();
|
ORDER BY timestamp ASC",
|
||||||
|
new { from, to }).AsList();
|
||||||
|
|
||||||
// For each cost entry, walk up parentId chain to find the nearest user message.
|
// Map chainId to sequential index (for coloring)
|
||||||
// That user message ID becomes the chain root.
|
|
||||||
// Entries sharing the same user-message ancestor = same turn/chain.
|
|
||||||
var chainRoot = new Dictionary<string, string>();
|
|
||||||
|
|
||||||
string FindUserRoot(string entryId, SqliteConnection conn)
|
|
||||||
{
|
|
||||||
if (chainRoot.TryGetValue(entryId, out var cached)) return cached;
|
|
||||||
|
|
||||||
var visited = new HashSet<string> { entryId };
|
|
||||||
var currentId = entryId;
|
|
||||||
|
|
||||||
while (true)
|
|
||||||
{
|
|
||||||
var row = conn.QueryFirstOrDefault(
|
|
||||||
@"SELECT json_extract(data, '$.parentId') AS parentId,
|
|
||||||
json_extract(data, '$.message.role') AS role
|
|
||||||
FROM entries WHERE id = @id",
|
|
||||||
new { id = currentId });
|
|
||||||
|
|
||||||
if (row == null) break;
|
|
||||||
|
|
||||||
string role = row.role;
|
|
||||||
// If we landed on a user message, that's our chain root
|
|
||||||
if (role == "user" && currentId != entryId)
|
|
||||||
{
|
|
||||||
chainRoot[entryId] = currentId;
|
|
||||||
return currentId;
|
|
||||||
}
|
|
||||||
|
|
||||||
string parentId = row.parentId;
|
|
||||||
if (parentId == null || visited.Contains(parentId)) break;
|
|
||||||
visited.Add(parentId);
|
|
||||||
currentId = parentId;
|
|
||||||
}
|
|
||||||
|
|
||||||
// No user message found - entry is its own root
|
|
||||||
chainRoot[entryId] = entryId;
|
|
||||||
return entryId;
|
|
||||||
}
|
|
||||||
|
|
||||||
foreach (var e in costEntries) FindUserRoot((string)e.id, db);
|
|
||||||
|
|
||||||
// Assign chain index by first appearance
|
|
||||||
var chainIndex = new Dictionary<string, int>();
|
var chainIndex = new Dictionary<string, int>();
|
||||||
int nextChain = 0;
|
int nextChain = 0;
|
||||||
var result = costEntries.Select(e =>
|
var result = rows.Select(e =>
|
||||||
{
|
{
|
||||||
var root = chainRoot[(string)e.id];
|
string cid = e.chainId ?? e.id;
|
||||||
if (!chainIndex.ContainsKey(root)) chainIndex[root] = nextChain++;
|
if (!chainIndex.ContainsKey(cid)) chainIndex[cid] = nextChain++;
|
||||||
return new
|
return new
|
||||||
{
|
{
|
||||||
e.id, e.session, e.type, e.timestamp,
|
e.id, e.session, e.type, e.timestamp,
|
||||||
e.cost, e.inputTokens, e.outputTokens, e.model,
|
e.cost, e.inputTokens, e.outputTokens, e.model,
|
||||||
e.data,
|
e.data,
|
||||||
chainId = chainIndex[root]
|
chainId = chainIndex[cid]
|
||||||
};
|
};
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -235,12 +213,12 @@ app.MapGet("/api/entry-chains", (Func<SqliteConnection> factory, string from, st
|
|||||||
// POST /api/sync - run export.mjs to import new entries
|
// POST /api/sync - run export.mjs to import new entries
|
||||||
app.MapPost("/api/sync", async () =>
|
app.MapPost("/api/sync", async () =>
|
||||||
{
|
{
|
||||||
var scriptPath = Path.GetFullPath(Path.Combine(AppContext.BaseDirectory, "..", "..", "..", "..", "export.mjs"));
|
var wd = "/home/node/.openclaw/workspace/projects/token-analytics/jsonl-sync";
|
||||||
var psi = new System.Diagnostics.ProcessStartInfo("node", scriptPath)
|
var psi = new System.Diagnostics.ProcessStartInfo("docker", $"exec meleki node {wd}/export.mjs")
|
||||||
{
|
{
|
||||||
RedirectStandardOutput = true,
|
RedirectStandardOutput = true,
|
||||||
RedirectStandardError = true,
|
RedirectStandardError = true,
|
||||||
WorkingDirectory = Path.GetDirectoryName(scriptPath)!
|
WorkingDirectory = wd!
|
||||||
};
|
};
|
||||||
var proc = System.Diagnostics.Process.Start(psi)!;
|
var proc = System.Diagnostics.Process.Start(psi)!;
|
||||||
var stdout = await proc.StandardOutput.ReadToEndAsync();
|
var stdout = await proc.StandardOutput.ReadToEndAsync();
|
||||||
@@ -7,5 +7,5 @@
|
|||||||
},
|
},
|
||||||
"AllowedHosts": "*",
|
"AllowedHosts": "*",
|
||||||
"Urls": "http://0.0.0.0:18684",
|
"Urls": "http://0.0.0.0:18684",
|
||||||
"DbPath": "../sessions.db"
|
"DbPath": "jsonl-sync/sessions.db"
|
||||||
}
|
}
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
set -e
|
||||||
|
|
||||||
|
export PATH="$HOME/.dotnet:$PATH"
|
||||||
|
cd ~/.openclaw/workspace/projects/token-analytics
|
||||||
|
|
||||||
|
dotnet publish -c Release -o publish
|
||||||
|
|
||||||
|
|
||||||
|
ssh -o ConnectTimeout=5 meleki@192.168.100.200 "nohup sudo systemctl restart token-analytics > /dev/null 2>&1 &"
|
||||||
|
echo "Service restarted"
|
||||||
-81
@@ -1,81 +0,0 @@
|
|||||||
import fs from 'node:fs';
|
|
||||||
import path from 'node:path';
|
|
||||||
import { fileURLToPath } from 'node:url';
|
|
||||||
import Database from 'better-sqlite3';
|
|
||||||
|
|
||||||
const __dirname = path.dirname(fileURLToPath(import.meta.url));
|
|
||||||
const SESSIONS_DIR = '/home/node/.openclaw/agents/main/sessions';
|
|
||||||
const DB_FILE = path.join(__dirname, 'sessions.db');
|
|
||||||
|
|
||||||
// DB öffnen + Schema anlegen
|
|
||||||
const db = new Database(DB_FILE);
|
|
||||||
|
|
||||||
db.exec(`
|
|
||||||
CREATE TABLE IF NOT EXISTS entries (
|
|
||||||
id TEXT PRIMARY KEY,
|
|
||||||
session TEXT NOT NULL,
|
|
||||||
type TEXT,
|
|
||||||
timestamp TEXT,
|
|
||||||
data TEXT NOT NULL
|
|
||||||
);
|
|
||||||
|
|
||||||
CREATE INDEX IF NOT EXISTS idx_session ON entries (session);
|
|
||||||
CREATE INDEX IF NOT EXISTS idx_type ON entries (type);
|
|
||||||
CREATE INDEX IF NOT EXISTS idx_timestamp ON entries (timestamp);
|
|
||||||
|
|
||||||
CREATE TABLE IF NOT EXISTS state (
|
|
||||||
file TEXT PRIMARY KEY,
|
|
||||||
lines INTEGER NOT NULL DEFAULT 0
|
|
||||||
);
|
|
||||||
`);
|
|
||||||
|
|
||||||
const getState = db.prepare('SELECT lines FROM state WHERE file = ?');
|
|
||||||
const setState = db.prepare('INSERT OR REPLACE INTO state (file, lines) VALUES (?, ?)');
|
|
||||||
const insertEntry = db.prepare(`
|
|
||||||
INSERT OR IGNORE INTO entries (id, session, type, timestamp, data)
|
|
||||||
VALUES (@id, @session, @type, @timestamp, @data)
|
|
||||||
`);
|
|
||||||
|
|
||||||
// Find all JSONL files
|
|
||||||
const jsonlFiles = fs.readdirSync(SESSIONS_DIR)
|
|
||||||
.filter(f => f.endsWith('.jsonl') && !f.includes('.deleted'));
|
|
||||||
|
|
||||||
let exported = 0;
|
|
||||||
|
|
||||||
const runImport = db.transaction(() => {
|
|
||||||
for (const file of jsonlFiles) {
|
|
||||||
const filePath = path.join(SESSIONS_DIR, file);
|
|
||||||
const sessionName = file.replace('.jsonl', '');
|
|
||||||
const lines = fs.readFileSync(filePath, 'utf-8').split('\n').filter(l => l.trim());
|
|
||||||
|
|
||||||
const lastProcessed = getState.get(file)?.lines ?? 0;
|
|
||||||
|
|
||||||
for (let i = lastProcessed; i < lines.length; i++) {
|
|
||||||
try {
|
|
||||||
const entry = JSON.parse(lines[i]);
|
|
||||||
if (!entry.timestamp) continue;
|
|
||||||
|
|
||||||
const id = entry.id || `${sessionName}-${i}`;
|
|
||||||
|
|
||||||
insertEntry.run({
|
|
||||||
id,
|
|
||||||
session: sessionName,
|
|
||||||
type: entry.type ?? 'unknown',
|
|
||||||
timestamp: entry.timestamp,
|
|
||||||
data: JSON.stringify(entry),
|
|
||||||
});
|
|
||||||
|
|
||||||
exported++;
|
|
||||||
} catch {
|
|
||||||
// skip malformed lines
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
setState.run(file, lines.length);
|
|
||||||
}
|
|
||||||
});
|
|
||||||
|
|
||||||
runImport();
|
|
||||||
db.close();
|
|
||||||
|
|
||||||
console.log(`Exportiert: ${exported} neue Einträge → ${DB_FILE}`);
|
|
||||||
@@ -0,0 +1,140 @@
|
|||||||
|
import fs from 'node:fs';
|
||||||
|
import path from 'node:path';
|
||||||
|
import { fileURLToPath } from 'node:url';
|
||||||
|
import Database from 'better-sqlite3';
|
||||||
|
|
||||||
|
const __dirname = path.dirname(fileURLToPath(import.meta.url));
|
||||||
|
const SESSIONS_DIR = '/home/node/.openclaw/agents/main/sessions';
|
||||||
|
const DB_FILE = path.join(__dirname, 'sessions.db');
|
||||||
|
|
||||||
|
const db = new Database(DB_FILE);
|
||||||
|
|
||||||
|
// Schema
|
||||||
|
db.exec(`
|
||||||
|
CREATE TABLE IF NOT EXISTS entries (
|
||||||
|
id TEXT PRIMARY KEY,
|
||||||
|
session TEXT NOT NULL,
|
||||||
|
type TEXT,
|
||||||
|
timestamp TEXT,
|
||||||
|
data TEXT NOT NULL
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_session ON entries (session);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_type ON entries (type);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_timestamp ON entries (timestamp);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS state (
|
||||||
|
file TEXT PRIMARY KEY,
|
||||||
|
lines INTEGER NOT NULL DEFAULT 0
|
||||||
|
);
|
||||||
|
`);
|
||||||
|
|
||||||
|
// Add chainId column if missing (migration)
|
||||||
|
try { db.exec('ALTER TABLE entries ADD COLUMN chainId TEXT'); } catch {}
|
||||||
|
try { db.exec('CREATE INDEX IF NOT EXISTS idx_chainId ON entries (chainId)'); } catch {}
|
||||||
|
|
||||||
|
const getState = db.prepare('SELECT lines FROM state WHERE file = ?');
|
||||||
|
const setState = db.prepare('INSERT OR REPLACE INTO state (file, lines) VALUES (?, ?)');
|
||||||
|
const insertEntry = db.prepare(`
|
||||||
|
INSERT OR IGNORE INTO entries (id, session, type, timestamp, data)
|
||||||
|
VALUES (@id, @session, @type, @timestamp, @data)
|
||||||
|
`);
|
||||||
|
const getEntry = db.prepare('SELECT id, data FROM entries WHERE id = ?');
|
||||||
|
const setChainId = db.prepare('UPDATE entries SET chainId = @chainId WHERE id = @id');
|
||||||
|
const getUnchained = db.prepare(`
|
||||||
|
SELECT id, data FROM entries
|
||||||
|
WHERE chainId IS NULL
|
||||||
|
ORDER BY timestamp ASC
|
||||||
|
`);
|
||||||
|
|
||||||
|
// --- Step 1: Import new entries ---
|
||||||
|
const jsonlFiles = fs.readdirSync(SESSIONS_DIR)
|
||||||
|
.filter(f => f.endsWith('.jsonl') && !f.includes('.deleted'));
|
||||||
|
|
||||||
|
let exported = 0;
|
||||||
|
|
||||||
|
const runImport = db.transaction(() => {
|
||||||
|
for (const file of jsonlFiles) {
|
||||||
|
const filePath = path.join(SESSIONS_DIR, file);
|
||||||
|
const sessionName = file.replace('.jsonl', '');
|
||||||
|
const lines = fs.readFileSync(filePath, 'utf-8').split('\n').filter(l => l.trim());
|
||||||
|
const lastProcessed = getState.get(file)?.lines ?? 0;
|
||||||
|
|
||||||
|
for (let i = lastProcessed; i < lines.length; i++) {
|
||||||
|
try {
|
||||||
|
const entry = JSON.parse(lines[i]);
|
||||||
|
if (!entry.timestamp) continue;
|
||||||
|
const id = entry.id || `${sessionName}-${i}`;
|
||||||
|
insertEntry.run({
|
||||||
|
id,
|
||||||
|
session: sessionName,
|
||||||
|
type: entry.type ?? 'unknown',
|
||||||
|
timestamp: entry.timestamp,
|
||||||
|
data: JSON.stringify(entry),
|
||||||
|
});
|
||||||
|
exported++;
|
||||||
|
} catch { /* skip */ }
|
||||||
|
}
|
||||||
|
|
||||||
|
setState.run(file, lines.length);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
runImport();
|
||||||
|
|
||||||
|
// --- Step 2: Compute chainId for entries that don't have one ---
|
||||||
|
// chainId = the nearest user message when walking up the parentId chain.
|
||||||
|
// If no user message found, chainId = own id.
|
||||||
|
|
||||||
|
let chained = 0;
|
||||||
|
const chainCache = new Map(); // id -> chainId
|
||||||
|
|
||||||
|
function resolveChainId(id) {
|
||||||
|
if (chainCache.has(id)) return chainCache.get(id);
|
||||||
|
|
||||||
|
const visited = new Set();
|
||||||
|
let currentId = id;
|
||||||
|
|
||||||
|
while (currentId && !visited.has(currentId)) {
|
||||||
|
visited.add(currentId);
|
||||||
|
|
||||||
|
if (chainCache.has(currentId) && currentId !== id) {
|
||||||
|
// Already resolved - reuse
|
||||||
|
const result = chainCache.get(currentId);
|
||||||
|
chainCache.set(id, result);
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
const row = getEntry.get(currentId);
|
||||||
|
if (!row) break;
|
||||||
|
|
||||||
|
const d = JSON.parse(row.data);
|
||||||
|
|
||||||
|
// If this is a user message, it's the chain root
|
||||||
|
if (d.message?.role === 'user') {
|
||||||
|
chainCache.set(id, currentId);
|
||||||
|
return currentId;
|
||||||
|
}
|
||||||
|
|
||||||
|
currentId = d.parentId || null;
|
||||||
|
}
|
||||||
|
|
||||||
|
// No user message found - self is root
|
||||||
|
chainCache.set(id, id);
|
||||||
|
return id;
|
||||||
|
}
|
||||||
|
|
||||||
|
const runChaining = db.transaction(() => {
|
||||||
|
const unchained = getUnchained.all();
|
||||||
|
|
||||||
|
for (const row of unchained) {
|
||||||
|
const chainId = resolveChainId(row.id);
|
||||||
|
setChainId.run({ id: row.id, chainId });
|
||||||
|
chained++;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
runChaining();
|
||||||
|
db.close();
|
||||||
|
|
||||||
|
console.log(`Exportiert: ${exported} neue, ${chained} verkettet → ${DB_FILE}`);
|
||||||
File renamed without changes.
File renamed without changes.
@@ -1,23 +0,0 @@
|
|||||||
{
|
|
||||||
"$schema": "https://json.schemastore.org/launchsettings.json",
|
|
||||||
"profiles": {
|
|
||||||
"http": {
|
|
||||||
"commandName": "Project",
|
|
||||||
"dotnetRunMessages": true,
|
|
||||||
"launchBrowser": true,
|
|
||||||
"applicationUrl": "http://localhost:5169",
|
|
||||||
"environmentVariables": {
|
|
||||||
"ASPNETCORE_ENVIRONMENT": "Development"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"https": {
|
|
||||||
"commandName": "Project",
|
|
||||||
"dotnetRunMessages": true,
|
|
||||||
"launchBrowser": true,
|
|
||||||
"applicationUrl": "https://localhost:7160;http://localhost:5169",
|
|
||||||
"environmentVariables": {
|
|
||||||
"ASPNETCORE_ENVIRONMENT": "Development"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,9 +0,0 @@
|
|||||||
{
|
|
||||||
"DetailedErrors": true,
|
|
||||||
"Logging": {
|
|
||||||
"LogLevel": {
|
|
||||||
"Default": "Information",
|
|
||||||
"Microsoft.AspNetCore": "Warning"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -4,9 +4,14 @@
|
|||||||
<TargetFramework>net10.0</TargetFramework>
|
<TargetFramework>net10.0</TargetFramework>
|
||||||
<Nullable>enable</Nullable>
|
<Nullable>enable</Nullable>
|
||||||
<ImplicitUsings>enable</ImplicitUsings>
|
<ImplicitUsings>enable</ImplicitUsings>
|
||||||
<RootNamespace>session_analytics_web</RootNamespace>
|
<RootNamespace>token_analytics</RootNamespace>
|
||||||
</PropertyGroup>
|
</PropertyGroup>
|
||||||
|
|
||||||
|
<ItemGroup>
|
||||||
|
<Content Remove="publish/**" />
|
||||||
|
<None Remove="publish/**" />
|
||||||
|
</ItemGroup>
|
||||||
|
|
||||||
<ItemGroup>
|
<ItemGroup>
|
||||||
<PackageReference Include="Dapper" Version="2.1.66" />
|
<PackageReference Include="Dapper" Version="2.1.66" />
|
||||||
<PackageReference Include="Microsoft.Data.Sqlite" Version="10.0.3" />
|
<PackageReference Include="Microsoft.Data.Sqlite" Version="10.0.3" />
|
||||||
@@ -34,9 +34,31 @@ async function init() {
|
|||||||
viewFrom.setDate(viewFrom.getDate() - ZOOM_LEVELS[0].days);
|
viewFrom.setDate(viewFrom.getDate() - ZOOM_LEVELS[0].days);
|
||||||
zoomIndex = 0;
|
zoomIndex = 0;
|
||||||
|
|
||||||
|
restoreFromHash();
|
||||||
await loadView();
|
await loadView();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function saveToHash() {
|
||||||
|
const state = {
|
||||||
|
f: viewFrom.toISOString(),
|
||||||
|
t: viewTo.toISOString(),
|
||||||
|
z: zoomIndex,
|
||||||
|
e: entryMode ? 1 : 0,
|
||||||
|
};
|
||||||
|
location.hash = btoa(JSON.stringify(state));
|
||||||
|
}
|
||||||
|
|
||||||
|
function restoreFromHash() {
|
||||||
|
if (!location.hash || location.hash.length < 2) return;
|
||||||
|
try {
|
||||||
|
const state = JSON.parse(atob(location.hash.slice(1)));
|
||||||
|
viewFrom = new Date(state.f);
|
||||||
|
viewTo = new Date(state.t);
|
||||||
|
zoomIndex = state.z ?? 0;
|
||||||
|
entryMode = !!state.e;
|
||||||
|
} catch {}
|
||||||
|
}
|
||||||
|
|
||||||
// --- Main load ---
|
// --- Main load ---
|
||||||
async function loadView() {
|
async function loadView() {
|
||||||
const from = viewFrom.toISOString();
|
const from = viewFrom.toISOString();
|
||||||
@@ -63,10 +85,14 @@ async function loadView() {
|
|||||||
api(`/api/cost-summary?from=${enc(from)}&to=${enc(to)}`),
|
api(`/api/cost-summary?from=${enc(from)}&to=${enc(to)}`),
|
||||||
]);
|
]);
|
||||||
|
|
||||||
|
const chains = await api(`/api/cost-chains?from=${enc(from)}&to=${enc(to)}`);
|
||||||
|
|
||||||
renderSummary(summary);
|
renderSummary(summary);
|
||||||
renderBucketChart(buckets, bucketMinutes);
|
renderBucketChart(buckets, bucketMinutes);
|
||||||
renderBlocks(buckets, bucketMinutes);
|
renderChainBlocks(chains);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
saveToHash();
|
||||||
}
|
}
|
||||||
|
|
||||||
function renderSummary(s) {
|
function renderSummary(s) {
|
||||||
@@ -188,46 +214,42 @@ function renderEntryChart(entries) {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
// --- Blocks (normal zoom) ---
|
// --- Chain blocks (grouped by conversation chain, sorted by cost) ---
|
||||||
function renderBlocks(buckets, bucketMinutes) {
|
function renderChainBlocks(chains) {
|
||||||
const container = $('blocks');
|
const container = $('blocks');
|
||||||
container.innerHTML = '';
|
container.innerHTML = '';
|
||||||
|
|
||||||
const bucketMap = {};
|
if (!chains.length) {
|
||||||
buckets.forEach(b => { bucketMap[b.bucket] = b; });
|
container.innerHTML = '<p style="color:var(--muted);padding:40px;text-align:center">Keine Einträge mit Kosten in diesem Zeitraum.</p>';
|
||||||
|
return;
|
||||||
const sortedBuckets = [];
|
|
||||||
for (let i = 0; i < BAR_COUNT; i++) {
|
|
||||||
const b = bucketMap[i];
|
|
||||||
if (!b || b.cost === 0) continue;
|
|
||||||
sortedBuckets.push({ ...b, idx: i });
|
|
||||||
}
|
}
|
||||||
sortedBuckets.sort((a, b) => b.cost - a.cost);
|
|
||||||
|
|
||||||
for (const { idx: i, ...b } of sortedBuckets) {
|
|
||||||
const bucketStart = new Date(viewFrom.getTime() + i * bucketMinutes * 60000);
|
|
||||||
const bucketEnd = new Date(viewFrom.getTime() + (i + 1) * bucketMinutes * 60000);
|
|
||||||
|
|
||||||
|
chains.forEach((c, ci) => {
|
||||||
const block = document.createElement('div');
|
const block = document.createElement('div');
|
||||||
block.className = 'block';
|
block.className = 'block';
|
||||||
block.innerHTML = `
|
block.innerHTML = `
|
||||||
<div class="block-header">
|
<div class="block-header">
|
||||||
<span class="block-label">${fmtRange(bucketStart, bucketEnd)}</span>
|
<span class="block-label">${fmtShort(new Date(c.first))} - ${fmtShort(new Date(c.last))}</span>
|
||||||
<span class="block-stats">
|
<span class="block-stats">
|
||||||
<span class="block-count">${b.count} Einträge</span>
|
<span class="block-count">${c.count} Einträge</span>
|
||||||
<span class="block-cost">$${b.cost.toFixed(4)}</span>
|
<span class="block-cost">$${c.totalCost.toFixed(4)}</span>
|
||||||
</span>
|
</span>
|
||||||
</div>
|
</div>
|
||||||
<div class="block-entries" id="be-${i}"></div>
|
|
||||||
`;
|
`;
|
||||||
block.querySelector('.block-header').addEventListener('click', () =>
|
block.querySelector('.block-header').addEventListener('click', () => {
|
||||||
toggleBlock(i, bucketStart, bucketEnd));
|
// Jump directly to entry mode, centered on this chain
|
||||||
|
const first = new Date(c.first);
|
||||||
|
const last = new Date(c.last);
|
||||||
|
const pad = Math.max((last - first) * 0.5, 60000); // min 1 min padding
|
||||||
|
zoomStack.push({ from: new Date(viewFrom), to: new Date(viewTo), zoom: zoomIndex, entry: false });
|
||||||
|
viewFrom = new Date(first.getTime() - pad);
|
||||||
|
viewTo = new Date(last.getTime() + pad);
|
||||||
|
zoomIndex = ZOOM_LEVELS.length - 1;
|
||||||
|
entryMode = true;
|
||||||
|
loadView();
|
||||||
|
});
|
||||||
container.appendChild(block);
|
container.appendChild(block);
|
||||||
}
|
});
|
||||||
|
|
||||||
if (!container.children.length) {
|
|
||||||
container.innerHTML = '<p style="color:var(--muted);padding:40px;text-align:center">Keine Einträge mit Kosten in diesem Zeitraum.</p>';
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// --- Entry list (flat, sorted by cost) ---
|
// --- Entry list (flat, sorted by cost) ---
|
||||||
@@ -259,22 +281,29 @@ function renderEntryList(entries) {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
// --- Toggle block entries ---
|
// --- Toggle chain block entries ---
|
||||||
async function toggleBlock(idx, from, to) {
|
async function toggleChainBlock(idx, chainId) {
|
||||||
const el = $(`be-${idx}`);
|
const el = $(`chain-block-${idx}`);
|
||||||
if (el.classList.contains('open')) { el.classList.remove('open'); return; }
|
if (el.classList.contains('open')) { el.classList.remove('open'); return; }
|
||||||
|
|
||||||
if (!el.dataset.loaded) {
|
if (!el.dataset.loaded) {
|
||||||
const entries = await api(`/api/cost-entries?from=${enc(from.toISOString())}&to=${enc(to.toISOString())}`);
|
// Load the full chain via entry-chain endpoint (using chainId = user message id)
|
||||||
el.innerHTML = entries.map(e => `
|
const chain = await api(`/api/entry-chain/${encodeURIComponent(chainId)}`);
|
||||||
<div class="entry" data-json='${escAttr(e.data)}'>
|
el.innerHTML = chain.map(entry => {
|
||||||
<span class="entry-time">${fmtTime(e.timestamp)}</span>
|
const d = JSON.parse(entry.data);
|
||||||
<span class="entry-model">${e.model || '-'}</span>
|
const msg = d.message;
|
||||||
<span class="entry-tokens">${shortNum(e.inputTokens)}/${shortNum(e.outputTokens)}</span>
|
if (!msg) return '';
|
||||||
<span class="entry-cost">$${Number(e.cost).toFixed(4)}</span>
|
const cost = msg.usage?.cost?.total;
|
||||||
<span class="entry-type">${e.type}</span>
|
return `
|
||||||
</div>
|
<div class="entry" data-json='${escAttr(entry.data)}'>
|
||||||
`).join('');
|
<span class="entry-time">${fmtTime(entry.timestamp)}</span>
|
||||||
|
<span class="entry-model">${msg.model || '-'}</span>
|
||||||
|
<span class="entry-tokens">${shortNum(msg.usage?.input)}/${shortNum(msg.usage?.output)}</span>
|
||||||
|
<span class="entry-cost">${cost ? '$' + cost.toFixed(4) : '-'}</span>
|
||||||
|
<span class="entry-type">${msg.role || entry.type}</span>
|
||||||
|
</div>
|
||||||
|
`;
|
||||||
|
}).filter(Boolean).join('');
|
||||||
el.querySelectorAll('.entry').forEach(row => {
|
el.querySelectorAll('.entry').forEach(row => {
|
||||||
const eid = JSON.parse(row.dataset.json).id;
|
const eid = JSON.parse(row.dataset.json).id;
|
||||||
row.addEventListener('click', () => openModal(row.dataset.json, eid));
|
row.addEventListener('click', () => openModal(row.dataset.json, eid));
|
||||||
@@ -3,7 +3,7 @@
|
|||||||
<head>
|
<head>
|
||||||
<meta charset="UTF-8" />
|
<meta charset="UTF-8" />
|
||||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||||
<title>Session Cost Analytics</title>
|
<title>Token Analytics</title>
|
||||||
<link rel="stylesheet" href="style.css" />
|
<link rel="stylesheet" href="style.css" />
|
||||||
</head>
|
</head>
|
||||||
<body>
|
<body>
|
||||||
@@ -125,7 +125,7 @@ header h1 { font-size: 1.1rem; font-weight: 600; color: var(--accent); }
|
|||||||
.bar {
|
.bar {
|
||||||
width: 100%;
|
width: 100%;
|
||||||
min-width: 4px;
|
min-width: 4px;
|
||||||
background: var(--green);
|
background: #5b7a9d;
|
||||||
border-radius: 3px 3px 0 0;
|
border-radius: 3px 3px 0 0;
|
||||||
opacity: 0.7;
|
opacity: 0.7;
|
||||||
transition: opacity 0.15s, background 0.15s;
|
transition: opacity 0.15s, background 0.15s;
|
||||||
@@ -133,7 +133,7 @@ header h1 { font-size: 1.1rem; font-weight: 600; color: var(--accent); }
|
|||||||
|
|
||||||
.bar-wrapper:hover .bar {
|
.bar-wrapper:hover .bar {
|
||||||
opacity: 1;
|
opacity: 1;
|
||||||
background: #6ee7b7;
|
background: #7a9dc0;
|
||||||
}
|
}
|
||||||
|
|
||||||
.bar-tooltip {
|
.bar-tooltip {
|
||||||
@@ -147,7 +147,7 @@ header h1 { font-size: 1.1rem; font-weight: 600; color: var(--accent); }
|
|||||||
font-size: 0.7rem;
|
font-size: 0.7rem;
|
||||||
white-space: nowrap;
|
white-space: nowrap;
|
||||||
z-index: 10;
|
z-index: 10;
|
||||||
color: var(--green);
|
color: var(--text);
|
||||||
font-weight: 600;
|
font-weight: 600;
|
||||||
pointer-events: none;
|
pointer-events: none;
|
||||||
}
|
}
|
||||||
@@ -195,6 +195,7 @@ header h1 { font-size: 1.1rem; font-weight: 600; color: var(--accent); }
|
|||||||
}
|
}
|
||||||
|
|
||||||
.block-header:hover { background: #22263a; }
|
.block-header:hover { background: #22263a; }
|
||||||
|
.block-header .block-label { pointer-events: none; }
|
||||||
|
|
||||||
.block-label {
|
.block-label {
|
||||||
font-family: monospace;
|
font-family: monospace;
|
||||||
Reference in new issue
Block a user