mirror of
https://github.com/kjanat/livedash-node.git
synced 2026-02-13 16:15:43 +01:00
fix: resolve Prettier markdown code block parsing errors
- Fix syntax errors in skills markdown files (.github/skills, .opencode/skills) - Change typescript to tsx for code blocks with JSX - Replace ellipsis (...) in array examples with valid syntax - Separate CSS from TypeScript into distinct code blocks - Convert JavaScript object examples to valid JSON in docs - Fix enum definitions with proper comma separation
This commit is contained in:
@@ -7,6 +7,7 @@ Successfully refactored the session processing pipeline from a simple status-bas
|
||||
## Problems Solved
|
||||
|
||||
### Original Issues
|
||||
|
||||
1. **Inconsistent Status Tracking**: The old system used a simple enum on SessionImport that didn't properly track the multi-stage processing pipeline
|
||||
2. **Poor Error Visibility**: Error messages were buried in the SessionImport table and not easily accessible
|
||||
3. **No Stage-Specific Tracking**: The system couldn't track which specific stage of processing failed
|
||||
@@ -14,32 +15,40 @@ Successfully refactored the session processing pipeline from a simple status-bas
|
||||
5. **Linting Errors**: Multiple TypeScript files referencing removed database fields
|
||||
|
||||
### Schema Changes Made
|
||||
|
||||
- **Removed** old `status`, `errorMsg`, and `processedAt` columns from SessionImport
|
||||
- **Removed** `processed` field from Session
|
||||
- **Removed** `processed` field from Session
|
||||
- **Added** new `SessionProcessingStatus` table with granular stage tracking
|
||||
- **Added** `ProcessingStage` and `ProcessingStatus` enums
|
||||
|
||||
## New Processing Pipeline
|
||||
|
||||
### Processing Stages
|
||||
|
||||
```typescript
|
||||
enum ProcessingStage {
|
||||
CSV_IMPORT // SessionImport created
|
||||
TRANSCRIPT_FETCH // Transcript content fetched
|
||||
SESSION_CREATION // Session + Messages created
|
||||
AI_ANALYSIS // AI processing completed
|
||||
QUESTION_EXTRACTION // Questions extracted
|
||||
CSV_IMPORT, // SessionImport created
|
||||
TRANSCRIPT_FETCH, // Transcript content fetched
|
||||
SESSION_CREATION, // Session + Messages created
|
||||
AI_ANALYSIS, // AI processing completed
|
||||
QUESTION_EXTRACTION, // Questions extracted
|
||||
}
|
||||
|
||||
enum ProcessingStatus {
|
||||
PENDING, IN_PROGRESS, COMPLETED, FAILED, SKIPPED
|
||||
PENDING,
|
||||
IN_PROGRESS,
|
||||
COMPLETED,
|
||||
FAILED,
|
||||
SKIPPED,
|
||||
}
|
||||
```
|
||||
|
||||
### Key Components
|
||||
|
||||
#### 1. ProcessingStatusManager
|
||||
|
||||
Centralized class for managing processing status with methods:
|
||||
|
||||
- `initializeSession()` - Set up processing status for new sessions
|
||||
- `startStage()`, `completeStage()`, `failStage()`, `skipStage()` - Stage management
|
||||
- `getSessionsNeedingProcessing()` - Query sessions by stage and status
|
||||
@@ -48,12 +57,14 @@ Centralized class for managing processing status with methods:
|
||||
- `resetStageForRetry()` - Reset failed stages
|
||||
|
||||
#### 2. Updated Processing Scheduler
|
||||
|
||||
- Integrated with new `ProcessingStatusManager`
|
||||
- Tracks AI analysis and question extraction stages
|
||||
- Records detailed processing metadata
|
||||
- Proper error handling and retry capabilities
|
||||
|
||||
#### 3. Migration System
|
||||
|
||||
- Successfully migrated all 109 existing sessions
|
||||
- Determined current state based on existing data
|
||||
- Preserved all existing functionality
|
||||
@@ -61,8 +72,9 @@ Centralized class for managing processing status with methods:
|
||||
## Current Pipeline Status
|
||||
|
||||
After migration and refactoring:
|
||||
|
||||
- **CSV_IMPORT**: 109 completed
|
||||
- **TRANSCRIPT_FETCH**: 109 completed
|
||||
- **TRANSCRIPT_FETCH**: 109 completed
|
||||
- **SESSION_CREATION**: 109 completed
|
||||
- **AI_ANALYSIS**: 16 completed, 93 pending
|
||||
- **QUESTION_EXTRACTION**: 11 completed, 98 pending
|
||||
@@ -70,18 +82,21 @@ After migration and refactoring:
|
||||
## Files Updated/Created
|
||||
|
||||
### New Files
|
||||
|
||||
- `lib/processingStatusManager.ts` - Core processing status management
|
||||
- `check-refactored-pipeline-status.ts` - New pipeline status checker
|
||||
- `migrate-to-refactored-system.ts` - Migration script
|
||||
- `docs/processing-system-refactor.md` - This documentation
|
||||
|
||||
### Updated Files
|
||||
|
||||
- `prisma/schema.prisma` - Added new processing status tables
|
||||
- `lib/processingScheduler.ts` - Integrated with new status system
|
||||
- `debug-import-status.ts` - Updated to use new system
|
||||
- `fix-import-status.ts` - Updated to use new system
|
||||
|
||||
### Removed Files
|
||||
|
||||
- `check-pipeline-status.ts` - Replaced by refactored version
|
||||
|
||||
## Benefits Achieved
|
||||
@@ -97,21 +112,25 @@ After migration and refactoring:
|
||||
## Usage Examples
|
||||
|
||||
### Check Pipeline Status
|
||||
|
||||
```bash
|
||||
npx tsx check-refactored-pipeline-status.ts
|
||||
```
|
||||
|
||||
### Debug Processing Issues
|
||||
|
||||
```bash
|
||||
npx tsx debug-import-status.ts
|
||||
```
|
||||
|
||||
### Fix/Retry Failed Sessions
|
||||
|
||||
```bash
|
||||
npx tsx fix-import-status.ts
|
||||
```
|
||||
|
||||
### Process Sessions
|
||||
|
||||
```bash
|
||||
npx tsx test-ai-processing.ts
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user