mirror of
https://github.com/kjanat/livedash-node.git
synced 2026-02-13 11:29:31 +01:00
fix: resolve Prettier markdown code block parsing errors
- Fix syntax errors in skills markdown files (.github/skills, .opencode/skills) - Change typescript to tsx for code blocks with JSX - Replace ellipsis (...) in array examples with valid syntax - Separate CSS from TypeScript into distinct code blocks - Convert JavaScript object examples to valid JSON in docs - Fix enum definitions with proper comma separation
This commit is contained in:
@@ -5,6 +5,7 @@ This document provides specific recommendations for optimizing database connecti
|
||||
## Current Issues Observed
|
||||
|
||||
From your logs, we can see:
|
||||
|
||||
```
|
||||
Can't reach database server at `ep-tiny-math-a2zsshve-pooler.eu-central-1.aws.neon.tech:5432`
|
||||
[NODE-CRON] [WARN] missed execution at Sun Jun 29 2025 12:00:00 GMT+0200! Possible blocking IO or high CPU
|
||||
@@ -13,16 +14,19 @@ Can't reach database server at `ep-tiny-math-a2zsshve-pooler.eu-central-1.aws.ne
|
||||
## Root Causes
|
||||
|
||||
### 1. Neon Connection Limits
|
||||
|
||||
- **Free Tier**: 20 concurrent connections
|
||||
- **Pro Tier**: 100 concurrent connections
|
||||
- **Pro Tier**: 100 concurrent connections
|
||||
- **Multiple schedulers** can quickly exhaust connections
|
||||
|
||||
### 2. Connection Pooling Issues
|
||||
|
||||
- Each scheduler was creating separate PrismaClient instances
|
||||
- No connection reuse between operations
|
||||
- No retry logic for temporary failures
|
||||
|
||||
### 3. Neon-Specific Challenges
|
||||
|
||||
- **Auto-pause**: Databases pause after inactivity
|
||||
- **Cold starts**: First connection after pause takes longer
|
||||
- **Regional latency**: eu-central-1 may have variable latency
|
||||
@@ -30,6 +34,7 @@ Can't reach database server at `ep-tiny-math-a2zsshve-pooler.eu-central-1.aws.ne
|
||||
## Solutions Implemented
|
||||
|
||||
### 1. Fixed Multiple PrismaClient Instances ✅
|
||||
|
||||
```typescript
|
||||
// Before: Each file created its own client
|
||||
const prisma = new PrismaClient(); // ❌
|
||||
@@ -39,30 +44,30 @@ import { prisma } from "./prisma.js"; // ✅
|
||||
```
|
||||
|
||||
### 2. Added Connection Retry Logic ✅
|
||||
|
||||
```typescript
|
||||
// Automatic retry for connection errors
|
||||
await withRetry(
|
||||
async () => await databaseOperation(),
|
||||
{
|
||||
maxRetries: 3,
|
||||
initialDelay: 2000,
|
||||
maxDelay: 10000,
|
||||
backoffMultiplier: 2,
|
||||
}
|
||||
);
|
||||
await withRetry(async () => await databaseOperation(), {
|
||||
maxRetries: 3,
|
||||
initialDelay: 2000,
|
||||
maxDelay: 10000,
|
||||
backoffMultiplier: 2,
|
||||
});
|
||||
```
|
||||
|
||||
### 3. Enhanced Connection Pooling ✅
|
||||
|
||||
```typescript
|
||||
// Production-ready pooling with @prisma/adapter-pg
|
||||
USE_ENHANCED_POOLING=true
|
||||
DATABASE_CONNECTION_LIMIT=20
|
||||
DATABASE_POOL_TIMEOUT=10
|
||||
USE_ENHANCED_POOLING = true;
|
||||
DATABASE_CONNECTION_LIMIT = 20;
|
||||
DATABASE_POOL_TIMEOUT = 10;
|
||||
```
|
||||
|
||||
## Neon-Specific Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
```bash
|
||||
# Optimized for Neon
|
||||
DATABASE_URL="postgresql://user:pass@ep-tiny-math-a2zsshve-pooler.eu-central-1.aws.neon.tech:5432/db?sslmode=require&connection_limit=15"
|
||||
@@ -79,6 +84,7 @@ SESSION_PROCESSING_INTERVAL="0 */2 * * *" # Every 2 hours instead of 1
|
||||
```
|
||||
|
||||
### Connection String Optimization
|
||||
|
||||
```bash
|
||||
# Add these parameters to your DATABASE_URL
|
||||
?sslmode=require # Required for Neon
|
||||
@@ -91,6 +97,7 @@ SESSION_PROCESSING_INTERVAL="0 */2 * * *" # Every 2 hours instead of 1
|
||||
## Monitoring & Troubleshooting
|
||||
|
||||
### 1. Health Check Endpoint
|
||||
|
||||
```bash
|
||||
# Check connection health
|
||||
curl -H "Authorization: Bearer your-token" \
|
||||
@@ -98,11 +105,13 @@ curl -H "Authorization: Bearer your-token" \
|
||||
```
|
||||
|
||||
### 2. Neon Dashboard Monitoring
|
||||
|
||||
- Monitor "Active connections" in Neon dashboard
|
||||
- Check for connection spikes during scheduler runs
|
||||
- Review query performance and slow queries
|
||||
|
||||
### 3. Application Logs
|
||||
|
||||
```bash
|
||||
# Look for connection patterns
|
||||
grep "Database connection" logs/*.log
|
||||
@@ -113,65 +122,77 @@ grep "retry" logs/*.log
|
||||
## Performance Optimizations
|
||||
|
||||
### 1. Reduce Scheduler Frequency
|
||||
|
||||
```typescript
|
||||
// Current intervals may be too aggressive
|
||||
CSV_IMPORT_INTERVAL="*/15 * * * *" // ➜ "*/30 * * * *"
|
||||
IMPORT_PROCESSING_INTERVAL="*/5 * * * *" // ➜ "*/10 * * * *"
|
||||
SESSION_PROCESSING_INTERVAL="0 * * * *" // ➜ "0 */2 * * *"
|
||||
CSV_IMPORT_INTERVAL = "*/15 * * * *"; // ➜ "*/30 * * * *"
|
||||
IMPORT_PROCESSING_INTERVAL = "*/5 * * * *"; // ➜ "*/10 * * * *"
|
||||
SESSION_PROCESSING_INTERVAL = "0 * * * *"; // ➜ "0 */2 * * *"
|
||||
```
|
||||
|
||||
### 2. Batch Size Optimization
|
||||
|
||||
```typescript
|
||||
// Reduce batch sizes to avoid long-running transactions
|
||||
CSV_IMPORT_BATCH_SIZE=50 // ➜ 25
|
||||
IMPORT_PROCESSING_BATCH_SIZE=50 // ➜ 25
|
||||
SESSION_PROCESSING_BATCH_SIZE=20 // ➜ 10
|
||||
CSV_IMPORT_BATCH_SIZE = 50; // ➜ 25
|
||||
IMPORT_PROCESSING_BATCH_SIZE = 50; // ➜ 25
|
||||
SESSION_PROCESSING_BATCH_SIZE = 20; // ➜ 10
|
||||
```
|
||||
|
||||
### 3. Connection Keepalive
|
||||
|
||||
```typescript
|
||||
// Keep connections warm to avoid cold starts
|
||||
const prisma = new PrismaClient({
|
||||
datasources: {
|
||||
db: {
|
||||
url: process.env.DATABASE_URL + "&keepalive=true"
|
||||
}
|
||||
}
|
||||
url: process.env.DATABASE_URL + "&keepalive=true",
|
||||
},
|
||||
},
|
||||
});
|
||||
```
|
||||
|
||||
## Troubleshooting Common Issues
|
||||
|
||||
### "Can't reach database server"
|
||||
|
||||
**Causes:**
|
||||
|
||||
- Neon database auto-paused
|
||||
- Connection limit exceeded
|
||||
- Network issues
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Enable enhanced pooling: `USE_ENHANCED_POOLING=true`
|
||||
2. Reduce connection limit: `DATABASE_CONNECTION_LIMIT=15`
|
||||
3. Implement retry logic (already done)
|
||||
4. Check Neon dashboard for database status
|
||||
|
||||
### "Connection terminated"
|
||||
|
||||
**Causes:**
|
||||
|
||||
- Idle connection timeout
|
||||
- Neon maintenance
|
||||
- Long-running transactions
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Increase pool timeout: `DATABASE_POOL_TIMEOUT=30`
|
||||
2. Add connection cycling
|
||||
3. Break large operations into smaller batches
|
||||
|
||||
### "Missed cron execution"
|
||||
|
||||
**Causes:**
|
||||
|
||||
- Blocking database operations
|
||||
- Scheduler overlap
|
||||
- High CPU usage
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Reduce scheduler frequency
|
||||
2. Add concurrency limits
|
||||
3. Monitor scheduler execution time
|
||||
@@ -179,6 +200,7 @@ const prisma = new PrismaClient({
|
||||
## Recommended Production Settings
|
||||
|
||||
### For Neon Free Tier (20 connections)
|
||||
|
||||
```bash
|
||||
DATABASE_CONNECTION_LIMIT=15
|
||||
DATABASE_POOL_TIMEOUT=30
|
||||
@@ -189,6 +211,7 @@ SESSION_PROCESSING_INTERVAL="0 */3 * * *"
|
||||
```
|
||||
|
||||
### For Neon Pro Tier (100 connections)
|
||||
|
||||
```bash
|
||||
DATABASE_CONNECTION_LIMIT=50
|
||||
DATABASE_POOL_TIMEOUT=20
|
||||
@@ -213,4 +236,4 @@ SESSION_PROCESSING_INTERVAL="0 */2 * * *"
|
||||
- [ ] Test health endpoint regularly
|
||||
- [ ] Set up alerts for connection failures
|
||||
|
||||
With these optimizations, your Neon database connections should be much more stable and efficient!
|
||||
With these optimizations, your Neon database connections should be much more stable and efficient!
|
||||
|
||||
@@ -7,6 +7,7 @@ Successfully refactored the session processing pipeline from a simple status-bas
|
||||
## Problems Solved
|
||||
|
||||
### Original Issues
|
||||
|
||||
1. **Inconsistent Status Tracking**: The old system used a simple enum on SessionImport that didn't properly track the multi-stage processing pipeline
|
||||
2. **Poor Error Visibility**: Error messages were buried in the SessionImport table and not easily accessible
|
||||
3. **No Stage-Specific Tracking**: The system couldn't track which specific stage of processing failed
|
||||
@@ -14,32 +15,40 @@ Successfully refactored the session processing pipeline from a simple status-bas
|
||||
5. **Linting Errors**: Multiple TypeScript files referencing removed database fields
|
||||
|
||||
### Schema Changes Made
|
||||
|
||||
- **Removed** old `status`, `errorMsg`, and `processedAt` columns from SessionImport
|
||||
- **Removed** `processed` field from Session
|
||||
- **Removed** `processed` field from Session
|
||||
- **Added** new `SessionProcessingStatus` table with granular stage tracking
|
||||
- **Added** `ProcessingStage` and `ProcessingStatus` enums
|
||||
|
||||
## New Processing Pipeline
|
||||
|
||||
### Processing Stages
|
||||
|
||||
```typescript
|
||||
enum ProcessingStage {
|
||||
CSV_IMPORT // SessionImport created
|
||||
TRANSCRIPT_FETCH // Transcript content fetched
|
||||
SESSION_CREATION // Session + Messages created
|
||||
AI_ANALYSIS // AI processing completed
|
||||
QUESTION_EXTRACTION // Questions extracted
|
||||
CSV_IMPORT, // SessionImport created
|
||||
TRANSCRIPT_FETCH, // Transcript content fetched
|
||||
SESSION_CREATION, // Session + Messages created
|
||||
AI_ANALYSIS, // AI processing completed
|
||||
QUESTION_EXTRACTION, // Questions extracted
|
||||
}
|
||||
|
||||
enum ProcessingStatus {
|
||||
PENDING, IN_PROGRESS, COMPLETED, FAILED, SKIPPED
|
||||
PENDING,
|
||||
IN_PROGRESS,
|
||||
COMPLETED,
|
||||
FAILED,
|
||||
SKIPPED,
|
||||
}
|
||||
```
|
||||
|
||||
### Key Components
|
||||
|
||||
#### 1. ProcessingStatusManager
|
||||
|
||||
Centralized class for managing processing status with methods:
|
||||
|
||||
- `initializeSession()` - Set up processing status for new sessions
|
||||
- `startStage()`, `completeStage()`, `failStage()`, `skipStage()` - Stage management
|
||||
- `getSessionsNeedingProcessing()` - Query sessions by stage and status
|
||||
@@ -48,12 +57,14 @@ Centralized class for managing processing status with methods:
|
||||
- `resetStageForRetry()` - Reset failed stages
|
||||
|
||||
#### 2. Updated Processing Scheduler
|
||||
|
||||
- Integrated with new `ProcessingStatusManager`
|
||||
- Tracks AI analysis and question extraction stages
|
||||
- Records detailed processing metadata
|
||||
- Proper error handling and retry capabilities
|
||||
|
||||
#### 3. Migration System
|
||||
|
||||
- Successfully migrated all 109 existing sessions
|
||||
- Determined current state based on existing data
|
||||
- Preserved all existing functionality
|
||||
@@ -61,8 +72,9 @@ Centralized class for managing processing status with methods:
|
||||
## Current Pipeline Status
|
||||
|
||||
After migration and refactoring:
|
||||
|
||||
- **CSV_IMPORT**: 109 completed
|
||||
- **TRANSCRIPT_FETCH**: 109 completed
|
||||
- **TRANSCRIPT_FETCH**: 109 completed
|
||||
- **SESSION_CREATION**: 109 completed
|
||||
- **AI_ANALYSIS**: 16 completed, 93 pending
|
||||
- **QUESTION_EXTRACTION**: 11 completed, 98 pending
|
||||
@@ -70,18 +82,21 @@ After migration and refactoring:
|
||||
## Files Updated/Created
|
||||
|
||||
### New Files
|
||||
|
||||
- `lib/processingStatusManager.ts` - Core processing status management
|
||||
- `check-refactored-pipeline-status.ts` - New pipeline status checker
|
||||
- `migrate-to-refactored-system.ts` - Migration script
|
||||
- `docs/processing-system-refactor.md` - This documentation
|
||||
|
||||
### Updated Files
|
||||
|
||||
- `prisma/schema.prisma` - Added new processing status tables
|
||||
- `lib/processingScheduler.ts` - Integrated with new status system
|
||||
- `debug-import-status.ts` - Updated to use new system
|
||||
- `fix-import-status.ts` - Updated to use new system
|
||||
|
||||
### Removed Files
|
||||
|
||||
- `check-pipeline-status.ts` - Replaced by refactored version
|
||||
|
||||
## Benefits Achieved
|
||||
@@ -97,21 +112,25 @@ After migration and refactoring:
|
||||
## Usage Examples
|
||||
|
||||
### Check Pipeline Status
|
||||
|
||||
```bash
|
||||
npx tsx check-refactored-pipeline-status.ts
|
||||
```
|
||||
|
||||
### Debug Processing Issues
|
||||
|
||||
```bash
|
||||
npx tsx debug-import-status.ts
|
||||
```
|
||||
|
||||
### Fix/Retry Failed Sessions
|
||||
|
||||
```bash
|
||||
npx tsx fix-import-status.ts
|
||||
```
|
||||
|
||||
### Process Sessions
|
||||
|
||||
```bash
|
||||
npx tsx test-ai-processing.ts
|
||||
```
|
||||
|
||||
@@ -9,22 +9,26 @@ The LiveDash system has two main schedulers that work together to fetch and proc
|
||||
|
||||
## Current Status (as of latest check)
|
||||
|
||||
- **Total sessions**: 107
|
||||
- **Processed sessions**: 0
|
||||
- **Sessions with transcript**: 0
|
||||
- **Ready for processing**: 0
|
||||
- **Total sessions**: 107
|
||||
- **Processed sessions**: 0
|
||||
- **Sessions with transcript**: 0
|
||||
- **Ready for processing**: 0
|
||||
|
||||
## How the `processed` Field Works
|
||||
|
||||
The ProcessingScheduler picks up sessions where `processed` is **NOT** `true`, which includes:
|
||||
|
||||
- `processed = false`
|
||||
- `processed = null`
|
||||
- `processed = false`
|
||||
- `processed = null`
|
||||
|
||||
**Query used:**
|
||||
|
||||
```javascript
|
||||
{ processed: { not: true } } // Either false or null
|
||||
{
|
||||
processed: {
|
||||
not: true;
|
||||
}
|
||||
} // Either false or null
|
||||
```
|
||||
|
||||
## Complete Workflow
|
||||
@@ -33,10 +37,10 @@ The ProcessingScheduler picks up sessions where `processed` is **NOT** `true`, w
|
||||
|
||||
**What it does:**
|
||||
|
||||
- Fetches session data from company CSV URLs
|
||||
- Creates session records in database with basic metadata
|
||||
- Sets `transcriptContent = null` initially
|
||||
- Sets `processed = null` initially
|
||||
- Fetches session data from company CSV URLs
|
||||
- Creates session records in database with basic metadata
|
||||
- Sets `transcriptContent = null` initially
|
||||
- Sets `processed = null` initially
|
||||
|
||||
**Runs:** Every 30 minutes (cron: `*/30 * * * *`)
|
||||
|
||||
@@ -44,9 +48,9 @@ The ProcessingScheduler picks up sessions where `processed` is **NOT** `true`, w
|
||||
|
||||
**What it does:**
|
||||
|
||||
- Downloads full transcript content for sessions
|
||||
- Updates `transcriptContent` field with actual conversation data
|
||||
- Sessions remain `processed = null` until AI processing
|
||||
- Downloads full transcript content for sessions
|
||||
- Updates `transcriptContent` field with actual conversation data
|
||||
- Sessions remain `processed = null` until AI processing
|
||||
|
||||
**Runs:** As part of session refresh process
|
||||
|
||||
@@ -54,11 +58,11 @@ The ProcessingScheduler picks up sessions where `processed` is **NOT** `true`, w
|
||||
|
||||
**What it does:**
|
||||
|
||||
- Finds sessions with transcript content where `processed != true`
|
||||
- Sends transcripts to OpenAI for analysis
|
||||
- Extracts: sentiment, category, questions, summary, etc.
|
||||
- Updates session with processed data
|
||||
- Sets `processed = true`
|
||||
- Finds sessions with transcript content where `processed != true`
|
||||
- Sends transcripts to OpenAI for analysis
|
||||
- Extracts: sentiment, category, questions, summary, etc.
|
||||
- Updates session with processed data
|
||||
- Sets `processed = true`
|
||||
|
||||
**Runs:** Every hour (cron: `0 * * * *`)
|
||||
|
||||
@@ -94,68 +98,68 @@ node scripts/manual-triggers.js both
|
||||
|
||||
1. **Check if sessions have transcripts:**
|
||||
|
||||
```bash
|
||||
node scripts/manual-triggers.js status
|
||||
```
|
||||
```bash
|
||||
node scripts/manual-triggers.js status
|
||||
```
|
||||
|
||||
2. **If "Sessions with transcript" is 0:**
|
||||
- Sessions exist but transcripts haven't been fetched yet
|
||||
- Run session refresh: `node scripts/manual-triggers.js refresh`
|
||||
|
||||
- Sessions exist but transcripts haven't been fetched yet
|
||||
- Run session refresh: `node scripts/manual-triggers.js refresh`
|
||||
|
||||
3. **If "Ready for processing" is 0 but "Sessions with transcript" > 0:**
|
||||
- All sessions with transcripts have already been processed
|
||||
- Check if `OPENAI_API_KEY` is set in environment
|
||||
|
||||
- All sessions with transcripts have already been processed
|
||||
- Check if `OPENAI_API_KEY` is set in environment
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### "No sessions found requiring processing"
|
||||
|
||||
- All sessions with transcripts have been processed (`processed = true`)
|
||||
- Or no sessions have transcript content yet
|
||||
- All sessions with transcripts have been processed (`processed = true`)
|
||||
- Or no sessions have transcript content yet
|
||||
|
||||
#### "OPENAI_API_KEY environment variable is not set"
|
||||
|
||||
- Add OpenAI API key to `.env.development` file
|
||||
- Restart the application
|
||||
- Add OpenAI API key to `.env.development` file
|
||||
- Restart the application
|
||||
|
||||
#### "Error fetching transcript: Unauthorized"
|
||||
|
||||
- CSV credentials are incorrect or expired
|
||||
- Check company CSV username/password in database
|
||||
- CSV credentials are incorrect or expired
|
||||
- Check company CSV username/password in database
|
||||
|
||||
## Database Field Mapping
|
||||
|
||||
### Before AI Processing
|
||||
|
||||
```javascript
|
||||
```json
|
||||
{
|
||||
id: "session-uuid",
|
||||
transcriptContent: "full conversation text" | null,
|
||||
processed: null,
|
||||
sentimentCategory: null,
|
||||
questions: null,
|
||||
summary: null,
|
||||
// ... other fields
|
||||
"id": "session-uuid",
|
||||
"transcriptContent": "full conversation text or null",
|
||||
"processed": null,
|
||||
"sentimentCategory": null,
|
||||
"questions": null,
|
||||
"summary": null
|
||||
}
|
||||
```
|
||||
|
||||
### After AI Processing
|
||||
|
||||
```javascript
|
||||
```json
|
||||
{
|
||||
id: "session-uuid",
|
||||
transcriptContent: "full conversation text",
|
||||
processed: true,
|
||||
sentimentCategory: "positive" | "neutral" | "negative",
|
||||
questions: '["question 1", "question 2"]', // JSON string
|
||||
summary: "Brief conversation summary",
|
||||
language: "en", // ISO 639-1 code
|
||||
messagesSent: 5,
|
||||
sentiment: 0.8, // Float value (-1 to 1)
|
||||
escalated: false,
|
||||
forwardedHr: false,
|
||||
category: "Schedule & Hours",
|
||||
// ... other fields
|
||||
"id": "session-uuid",
|
||||
"transcriptContent": "full conversation text",
|
||||
"processed": true,
|
||||
"sentimentCategory": "positive",
|
||||
"questions": "[\"question 1\", \"question 2\"]",
|
||||
"summary": "Brief conversation summary",
|
||||
"language": "en",
|
||||
"messagesSent": 5,
|
||||
"sentiment": 0.8,
|
||||
"escalated": false,
|
||||
"forwardedHr": false,
|
||||
"category": "Schedule & Hours"
|
||||
}
|
||||
```
|
||||
|
||||
@@ -163,16 +167,16 @@ node scripts/manual-triggers.js both
|
||||
|
||||
### Session Refresh Scheduler
|
||||
|
||||
- **File**: `lib/scheduler.js`
|
||||
- **Frequency**: Every 30 minutes
|
||||
- **Cron**: `*/30 * * * *`
|
||||
- **File**: `lib/scheduler.js`
|
||||
- **Frequency**: Every 30 minutes
|
||||
- **Cron**: `*/30 * * * *`
|
||||
|
||||
### Processing Scheduler
|
||||
### Processing Scheduler
|
||||
|
||||
- **File**: `lib/processingScheduler.js`
|
||||
- **Frequency**: Every hour
|
||||
- **Cron**: `0 * * * *`
|
||||
- **Batch size**: 10 sessions per run
|
||||
- **File**: `lib/processingScheduler.js`
|
||||
- **Frequency**: Every hour
|
||||
- **Cron**: `0 * * * *`
|
||||
- **Batch size**: 10 sessions per run
|
||||
|
||||
## Environment Variables Required
|
||||
|
||||
@@ -192,20 +196,20 @@ NEXTAUTH_URL="http://localhost:3000"
|
||||
|
||||
1. **Trigger session refresh** to fetch transcripts:
|
||||
|
||||
```bash
|
||||
node scripts/manual-triggers.js refresh
|
||||
```
|
||||
```bash
|
||||
node scripts/manual-triggers.js refresh
|
||||
```
|
||||
|
||||
2. **Check status** to see if transcripts were fetched:
|
||||
|
||||
```bash
|
||||
node scripts/manual-triggers.js status
|
||||
```
|
||||
```bash
|
||||
node scripts/manual-triggers.js status
|
||||
```
|
||||
|
||||
3. **Trigger processing** if transcripts are available:
|
||||
|
||||
```bash
|
||||
node scripts/manual-triggers.js process
|
||||
```
|
||||
```bash
|
||||
node scripts/manual-triggers.js process
|
||||
```
|
||||
|
||||
4. **View results** in the dashboard session details pages
|
||||
|
||||
@@ -25,8 +25,8 @@ CREATE INDEX Message_sessionId_order_idx ON Message(sessionId, order);
|
||||
|
||||
### Updated Session Table
|
||||
|
||||
- Added `messages` relation to Session model
|
||||
- Sessions can now have both raw transcript content AND parsed messages
|
||||
- Added `messages` relation to Session model
|
||||
- Sessions can now have both raw transcript content AND parsed messages
|
||||
|
||||
## New Components
|
||||
|
||||
@@ -46,35 +46,35 @@ export interface Message {
|
||||
|
||||
### 2. Transcript Parser (`lib/transcriptParser.js`)
|
||||
|
||||
- **`parseChatLogToJSON(logString)`** - Parses raw transcript text into structured messages
|
||||
- **`storeMessagesForSession(sessionId, messages)`** - Stores parsed messages in database
|
||||
- **`processTranscriptForSession(sessionId, transcriptContent)`** - Complete processing for one session
|
||||
- **`processAllUnparsedTranscripts()`** - Batch process all unparsed transcripts
|
||||
- **`getMessagesForSession(sessionId)`** - Retrieve messages for a session
|
||||
- **`parseChatLogToJSON(logString)`** - Parses raw transcript text into structured messages
|
||||
- **`storeMessagesForSession(sessionId, messages)`** - Stores parsed messages in database
|
||||
- **`processTranscriptForSession(sessionId, transcriptContent)`** - Complete processing for one session
|
||||
- **`processAllUnparsedTranscripts()`** - Batch process all unparsed transcripts
|
||||
- **`getMessagesForSession(sessionId)`** - Retrieve messages for a session
|
||||
|
||||
### 3. MessageViewer Component (`components/MessageViewer.tsx`)
|
||||
|
||||
- Chat-like interface for displaying parsed messages
|
||||
- Color-coded by role (User: blue, Assistant: gray, System: yellow)
|
||||
- Shows timestamps and message order
|
||||
- Scrollable with conversation metadata
|
||||
- Chat-like interface for displaying parsed messages
|
||||
- Color-coded by role (User: blue, Assistant: gray, System: yellow)
|
||||
- Shows timestamps and message order
|
||||
- Scrollable with conversation metadata
|
||||
|
||||
## Updated Components
|
||||
|
||||
### 1. Session API (`pages/api/dashboard/session/[id].ts`)
|
||||
|
||||
- Now includes parsed messages in session response
|
||||
- Messages are ordered by `order` field (ascending)
|
||||
- Now includes parsed messages in session response
|
||||
- Messages are ordered by `order` field (ascending)
|
||||
|
||||
### 2. Session Details Page (`app/dashboard/sessions/[id]/page.tsx`)
|
||||
|
||||
- Added MessageViewer component
|
||||
- Shows both parsed messages AND raw transcript
|
||||
- Prioritizes parsed messages when available
|
||||
- Added MessageViewer component
|
||||
- Shows both parsed messages AND raw transcript
|
||||
- Prioritizes parsed messages when available
|
||||
|
||||
### 3. ChatSession Interface (`lib/types.ts`)
|
||||
|
||||
- Added optional `messages?: Message[]` field
|
||||
- Added optional `messages?: Message[]` field
|
||||
|
||||
## Parsing Logic
|
||||
|
||||
@@ -90,11 +90,11 @@ The parser expects transcript format:
|
||||
|
||||
### Features
|
||||
|
||||
- **Multi-line support** - Messages can span multiple lines
|
||||
- **Timestamp parsing** - Converts DD.MM.YYYY HH:MM:SS to ISO format
|
||||
- **Role detection** - Extracts sender role from each message
|
||||
- **Ordering** - Maintains conversation order with explicit order field
|
||||
- **Sorting** - Messages sorted by timestamp, then by role (User before Assistant)
|
||||
- **Multi-line support** - Messages can span multiple lines
|
||||
- **Timestamp parsing** - Converts DD.MM.YYYY HH:MM:SS to ISO format
|
||||
- **Role detection** - Extracts sender role from each message
|
||||
- **Ordering** - Maintains conversation order with explicit order field
|
||||
- **Sorting** - Messages sorted by timestamp, then by role (User before Assistant)
|
||||
|
||||
## Manual Commands
|
||||
|
||||
@@ -113,8 +113,8 @@ node scripts/manual-triggers.js status
|
||||
|
||||
### Updated Commands
|
||||
|
||||
- **`status`** - Now shows transcript and parsing statistics
|
||||
- **`all`** - New command that runs refresh → parse → process in sequence
|
||||
- **`status`** - Now shows transcript and parsing statistics
|
||||
- **`all`** - New command that runs refresh → parse → process in sequence
|
||||
|
||||
## Workflow Integration
|
||||
|
||||
@@ -126,29 +126,35 @@ node scripts/manual-triggers.js status
|
||||
|
||||
### Database States
|
||||
|
||||
```javascript
|
||||
// After CSV fetch
|
||||
{
|
||||
transcriptContent: "raw text...",
|
||||
messages: [], // Empty
|
||||
processed: null
|
||||
}
|
||||
After CSV fetch:
|
||||
|
||||
// After parsing
|
||||
```json
|
||||
{
|
||||
transcriptContent: "raw text...",
|
||||
messages: [Message, Message, ...], // Parsed
|
||||
processed: null
|
||||
"transcriptContent": "raw text...",
|
||||
"messages": [],
|
||||
"processed": null
|
||||
}
|
||||
```
|
||||
|
||||
// After AI processing
|
||||
After parsing:
|
||||
|
||||
```json
|
||||
{
|
||||
transcriptContent: "raw text...",
|
||||
messages: [Message, Message, ...], // Parsed
|
||||
processed: true,
|
||||
sentimentCategory: "positive",
|
||||
summary: "Brief summary...",
|
||||
// ... other AI fields
|
||||
"transcriptContent": "raw text...",
|
||||
"messages": ["Message1", "Message2"],
|
||||
"processed": null
|
||||
}
|
||||
```
|
||||
|
||||
After AI processing:
|
||||
|
||||
```json
|
||||
{
|
||||
"transcriptContent": "raw text...",
|
||||
"messages": ["Message1", "Message2"],
|
||||
"processed": true,
|
||||
"sentimentCategory": "positive",
|
||||
"summary": "Brief summary..."
|
||||
}
|
||||
```
|
||||
|
||||
@@ -156,18 +162,18 @@ node scripts/manual-triggers.js status
|
||||
|
||||
### Before
|
||||
|
||||
- Only raw transcript text in a text area
|
||||
- Difficult to follow conversation flow
|
||||
- No clear distinction between speakers
|
||||
- Only raw transcript text in a text area
|
||||
- Difficult to follow conversation flow
|
||||
- No clear distinction between speakers
|
||||
|
||||
### After
|
||||
|
||||
- **Chat-like interface** with message bubbles
|
||||
- **Color-coded roles** for easy identification
|
||||
- **Timestamps** for each message
|
||||
- **Conversation metadata** (first/last message times)
|
||||
- **Fallback to raw transcript** if parsing fails
|
||||
- **Both views available** - structured AND raw
|
||||
- **Chat-like interface** with message bubbles
|
||||
- **Color-coded roles** for easy identification
|
||||
- **Timestamps** for each message
|
||||
- **Conversation metadata** (first/last message times)
|
||||
- **Fallback to raw transcript** if parsing fails
|
||||
- **Both views available** - structured AND raw
|
||||
|
||||
## Testing
|
||||
|
||||
@@ -195,34 +201,34 @@ node scripts/manual-triggers.js all
|
||||
|
||||
### Performance
|
||||
|
||||
- **Indexed queries** - Messages indexed by sessionId and order
|
||||
- **Efficient loading** - Only load messages when needed
|
||||
- **Cascading deletes** - Messages automatically deleted with sessions
|
||||
- **Indexed queries** - Messages indexed by sessionId and order
|
||||
- **Efficient loading** - Only load messages when needed
|
||||
- **Cascading deletes** - Messages automatically deleted with sessions
|
||||
|
||||
### Maintainability
|
||||
|
||||
- **Separation of concerns** - Parsing logic isolated in dedicated module
|
||||
- **Type safety** - Full TypeScript support for Message interface
|
||||
- **Error handling** - Graceful fallbacks when parsing fails
|
||||
- **Separation of concerns** - Parsing logic isolated in dedicated module
|
||||
- **Type safety** - Full TypeScript support for Message interface
|
||||
- **Error handling** - Graceful fallbacks when parsing fails
|
||||
|
||||
### Extensibility
|
||||
|
||||
- **Role flexibility** - Supports any role names (User, Assistant, System, etc.)
|
||||
- **Content preservation** - Multi-line messages fully supported
|
||||
- **Metadata ready** - Easy to add message-level metadata in future
|
||||
- **Role flexibility** - Supports any role names (User, Assistant, System, etc.)
|
||||
- **Content preservation** - Multi-line messages fully supported
|
||||
- **Metadata ready** - Easy to add message-level metadata in future
|
||||
|
||||
## Migration Notes
|
||||
|
||||
### Existing Data
|
||||
|
||||
- **No data loss** - Original transcript content preserved
|
||||
- **Backward compatibility** - Pages work with or without parsed messages
|
||||
- **Gradual migration** - Can parse transcripts incrementally
|
||||
- **No data loss** - Original transcript content preserved
|
||||
- **Backward compatibility** - Pages work with or without parsed messages
|
||||
- **Gradual migration** - Can parse transcripts incrementally
|
||||
|
||||
### Database Migration
|
||||
|
||||
- New Message table created with foreign key constraints
|
||||
- Existing Session table unchanged (only added relation)
|
||||
- Index created for efficient message queries
|
||||
- New Message table created with foreign key constraints
|
||||
- Existing Session table unchanged (only added relation)
|
||||
- Index created for efficient message queries
|
||||
|
||||
This implementation provides a solid foundation for enhanced conversation analysis and user experience while maintaining full backward compatibility.
|
||||
|
||||
Reference in New Issue
Block a user