kjanat/livegraphs-django
1
0
Fork 0
mirror of https://github.com/kjanat/livegraphs-django.git synced 2026-02-13 09:29:32 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity

fix: configure markdownlint-cli2 properly

Browse Source 
- Wrap config in 'config' key for markdownlint-cli2
- Use MD013 rule name instead of 'line-length' alias
- Disable MD013 line-length checks
- Add allowed languages: sh, python, csv, tree
- Fix broken link reference in TODO.md
- All markdown linting now passes (37 errors -> 0)
This commit is contained in:
Kaj Kowalski
2025-11-05 16:51:23 +01:00
parent 04705bdcb2
commit fe847a3d4e
9 changed files with 434 additions and 353 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
.editorconfig
View File

@@ -33,6 +33,7 @@ indent_size = 2
# Markdown files
[*.md]
trim_trailing_whitespace = false
indent_size = 4
# YAML files
[*.{yml,yaml}]

98
.github/copilot-instructions.md vendored
View File

@@ -2,44 +2,45 @@
## General Instructions
- Use clear and concise language.
- Provide code examples where applicable.
- Write clean code with Django best practices.
- Use comments to explain complex logic.
- Use packages and libraries where appropriate and possible to avoid reinventing the wheel.
- Update [TODO](TODO.md), [README](README.md) as fits.
- Use clear and concise language.
- Provide code examples where applicable.
- Write clean code with Django best practices.
- Use comments to explain complex logic.
- Use packages and libraries where appropriate and possible to avoid reinventing the wheel.
- Update [TODO](TODO.md), [README](README.md) as fits.
## uv
UV is a fast Python package and project manager written in Rust. Use UV to manage dependencies, virtual environments, and run Python scripts with improved performance.
UV is a fast Python package and project manager written in Rust. Use UV to manage dependencies, virtual environments,
and run Python scripts with improved performance.
### Running Python Scripts
- Execute a Python script with uv:
- Execute a Python script with uv:
```bash
uv run python ${FILE}.py
```
- Run a script with a specific Python version:
- Run a script with a specific Python version:
```bash
uv run python3.8 ${FILE}.py
```
- Run a script with arguments:
- Run a script with arguments:
```bash
uv run python ${FILE}.py --arg1 value1 --arg2 value2
```
- Add dependencies to standalone scripts:
- Add dependencies to standalone scripts:
```bash
uv add --script <package-name> ${FILE}.py
```
- Remove dependencies from a script:
- Remove dependencies from a script:
```bash
uv remove --script <package-name> ${FILE}.py
@@ -47,25 +48,25 @@ UV is a fast Python package and project manager written in Rust. Use UV to manag
### Package Management
- Install packages:
- Install packages:
```bash
uv pip install <package-name>
```
- Install from requirements file:
- Install from requirements file:
```bash
uv pip install -r requirements.txt
```
- Add a package to current project:
- Add a package to current project:
```bash
uv add <package-name>
```
- Remove a package:
- Remove a package:
```bash
uv remove <package-name>
@@ -73,20 +74,20 @@ UV is a fast Python package and project manager written in Rust. Use UV to manag
### Virtual Environment Management
- Create and activate a virtual environment:
- Create and activate a virtual environment:
```bash
uv venv .venv
source .venv/bin/activate # Linux/macOS
```
- Install project dependencies into an environment:
- Install project dependencies into an environment:
```bash
uv pip sync
```
- Lock dependencies for reproducible environments:
- Lock dependencies for reproducible environments:
```bash
uv lock
@@ -94,25 +95,25 @@ UV is a fast Python package and project manager written in Rust. Use UV to manag
### Project Management
- Create a new Python project:
- Create a new Python project:
```bash
uv init <project-name>
```
- Build a project into distribution archives:
- Build a project into distribution archives:
```bash
uv build
```
- View dependency tree:
- View dependency tree:
```bash
uv tree
```
- Publish package to PyPI:
- Publish package to PyPI:
```bash
uv publish
@@ -120,25 +121,25 @@ UV is a fast Python package and project manager written in Rust. Use UV to manag
### Python Version Management
- Install specific Python version:
- Install specific Python version:
```bash
uv python install 3.11
```
- List available Python versions:
- List available Python versions:
```bash
uv python list
```
- Find installed Python version:
- Find installed Python version:
```bash
uv python find
```
- Pin project to specific Python version:
- Pin project to specific Python version:
```bash
uv python pin 3.11
@@ -146,14 +147,15 @@ UV is a fast Python package and project manager written in Rust. Use UV to manag
### Performance Benefits
- UV offers significantly faster package installations than pip
- Built-in caching improves repeated operations
- Compatible with existing Python tooling ecosystem
- Reliable dependency resolution to avoid conflicts
- UV offers significantly faster package installations than pip
- Built-in caching improves repeated operations
- Compatible with existing Python tooling ecosystem
- Reliable dependency resolution to avoid conflicts
## Project Structure
This section provides a comprehensive overview of the LiveGraphsDjango project structure and the function of each key file. Please update this section whenever there are noteworthy changes to the structure or to a file's function.
This section provides a comprehensive overview of the LiveGraphsDjango project structure and the function of each key
file. Please update this section whenever there are noteworthy changes to the structure or to a file's function.
```tree
LiveGraphsDjango/
@@ -235,31 +237,31 @@ LiveGraphsDjango/
### Key Component Relationships
1. **Multi-Tenant Architecture**:
1. **Multi-Tenant Architecture**:
- Companies are the top-level organizational unit
- Users belong to Companies and have different permission levels
- DataSources are owned by Companies
- Dashboards display analytics based on DataSources
- Companies are the top-level organizational unit
- Users belong to Companies and have different permission levels
- DataSources are owned by Companies
- Dashboards display analytics based on DataSources
2. **Data Integration Flow**:
2. **Data Integration Flow**:
- External APIs are configured via ExternalDataSource models
- Data is fetched, parsed, and stored as ChatSessions and ChatMessages
- Dashboard views aggregate and visualize this data
- External APIs are configured via ExternalDataSource models
- Data is fetched, parsed, and stored as ChatSessions and ChatMessages
- Dashboard views aggregate and visualize this data
3. **Export Functionality**:
3. **Export Functionality**:
- Export available in CSV, JSON, and Excel formats
- Filtering options to customize exported data
- Export available in CSV, JSON, and Excel formats
- Filtering options to customize exported data
### Important Note
**Please update this section whenever:**
1. New files or directories are added to the project
2. The function of existing files changes significantly
3. New relationships between components are established
4. The architecture of the application changes
1. New files or directories are added to the project
2. The function of existing files changes significantly
3. New relationships between components are established
4. The architecture of the application changes
This ensures that anyone working with GitHub Copilot has an up-to-date understanding of the project structure.

73
.markdownlint-cli2.jsonc Normal file
View File

@@ -0,0 +1,73 @@
// Configuration for markdownlint-cli2
{
"config": {
"$schema": "https://cdn.jsdelivr.net/gh/DavidAnson/markdownlint@main/schema/markdownlint-config-schema.json",
"inline-html": false,
"code-block-style": {
"style": "fenced"
},
"code-fence-style": {
"style": "backtick"
},
"emphasis-style": {
"style": "asterisk"
},
"extended-ascii": {
"ascii-only": true
},
"fenced-code-language": {
"allowed_languages": [
"bash",
"sh",
"html",
"javascript",
"json",
"markdown",
"text",
"python",
"csv",
"tree"
],
"language_only": true
},
"heading-style": {
"style": "atx"
},
"hr-style": {
"style": "---"
},
"MD013": false,
"link-image-style": {
"collapsed": false,
"shortcut": false,
"url_inline": false
},
"no-duplicate-heading": {
"siblings_only": true
},
"ol-prefix": {
"style": "ordered"
},
"proper-names": {
"code_blocks": false,
"names": [
"Cake.Markdownlint",
"CommonMark",
"JavaScript",
"Markdown",
"markdown-it",
"markdownlint",
"Node.js"
]
},
"reference-links-images": {
"shortcut_syntax": true
},
"strong-style": {
"style": "asterisk"
},
"ul-style": {
"style": "dash"
}
}
}

6
.prettierrc → .prettierrc.json
View File

@@ -28,6 +28,12 @@
"proseWrap": "preserve",
"printWidth": 100
}
},
{
"files": ["*.jsonc"],
"options": {
"trailingComma": "none"
}
}
],
"plugins": ["prettier-plugin-jinja-template", "prettier-plugin-packagejson"]

435
README.md
View File

@@ -1,49 +1,52 @@
# Chat Analytics Dashboard
A Django application that creates an analytics dashboard for chat session data. The application allows different companies to have their own dashboards and view their own data.
A Django application that creates an analytics dashboard for chat session data. The application allows different
companies to have their own dashboards and view their own data.
## Project Overview
This Django project creates a multi-tenant dashboard application for analyzing chat session data. Companies can upload their chat data (in CSV format) and view analytics and metrics through an interactive dashboard. The application supports user authentication, role-based access control, and separate data isolation for different companies.
This Django project creates a multi-tenant dashboard application for analyzing chat session data. Companies can upload
their chat data (in CSV format) and view analytics and metrics through an interactive dashboard. The application
supports user authentication, role-based access control, and separate data isolation for different companies.
### Project Structure
The project consists of two main Django apps:
1. **accounts**: Handles user authentication, company management, and user roles
2. **dashboard**: Manages data sources, chat sessions, and dashboard visualization
3. **data_integration**: Handles external API data integration
1. **accounts**: Handles user authentication, company management, and user roles
2. **dashboard**: Manages data sources, chat sessions, and dashboard visualization
3. **data_integration**: Handles external API data integration
### Key Features
- **Multi-company Support**: Each company has their own private dashboards and data
- **User Management**: Different user roles (admin, company admin, regular user)
- **CSV File Upload**: Upload and process CSV files containing chat session data
- **Interactive Dashboard**: Visualize chat data with charts and metrics
- **Search Functionality**: Find specific chat sessions based on various criteria
- **Data Export**: Export data in CSV, JSON, and Excel formats
- **Data Exploration**: Drill down into individual chat sessions for detailed analysis
- **Responsive Design**: Mobile-friendly interface using Bootstrap 5
- **Multi-company Support**: Each company has their own private dashboards and data
- **User Management**: Different user roles (admin, company admin, regular user)
- **CSV File Upload**: Upload and process CSV files containing chat session data
- **Interactive Dashboard**: Visualize chat data with charts and metrics
- **Search Functionality**: Find specific chat sessions based on various criteria
- **Data Export**: Export data in CSV, JSON, and Excel formats
- **Data Exploration**: Drill down into individual chat sessions for detailed analysis
- **Responsive Design**: Mobile-friendly interface using Bootstrap 5
## Requirements
- Python 3.13+
- Django 5.2+
- UV package manager (recommended)
- Other dependencies listed in [`pyproject.toml`](./pyproject.toml)
- Python 3.13+
- Django 5.2+
- UV package manager (recommended)
- Other dependencies listed in [`pyproject.toml`](./pyproject.toml)
## Setup
### Local Development
1. Clone the repository:
1. Clone the repository:
```sh
git clone <repository-url>
cd LiveGraphsDjango
git clone https://github.com/kjanat/livegraphs-django.git
cd livegraphs-django
```
2. Install uv if you don't have it yet:
2. Install uv if you don't have it yet:
```sh
# Install using pip
@@ -56,14 +59,14 @@ The project consists of two main Django apps:
irm https://install.pypa.io/get-uv.ps1 | iex
```
3. Create a virtual environment and activate it:
3. Create a virtual environment and activate it:
```sh
uv venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
```
4. Install dependencies:
4. Install dependencies:
```sh
# Install all dependencies including dev dependencies
@@ -73,7 +76,7 @@ The project consists of two main Django apps:
uv pip install -e .
```
5. Run migrations:
5. Run migrations:
```sh
cd dashboard_project
@@ -81,13 +84,13 @@ The project consists of two main Django apps:
python manage.py migrate
```
6. Create a superuser:
6. Create a superuser:
```sh
python manage.py createsuperuser
```
7. Set up environment variables:
7. Set up environment variables:
```sh
# Copy the sample .env file
@@ -99,11 +102,11 @@ The project consists of two main Django apps:
Be sure to update:
- `EXTERNAL_API_USERNAME` and `EXTERNAL_API_PASSWORD` for the data integration API
- `DJANGO_SECRET_KEY` for production environments
- Redis URL if using a different configuration for Celery
- `EXTERNAL_API_USERNAME` and `EXTERNAL_API_PASSWORD` for the data integration API
- `DJANGO_SECRET_KEY` for production environments
- Redis URL if using a different configuration for Celery
8. Start Celery for background tasks:
8. Start Celery for background tasks:
```sh
# In a separate terminal
@@ -115,7 +118,7 @@ The project consists of two main Django apps:
celery -A dashboard_project beat --scheduler django_celery_beat.schedulers:DatabaseScheduler
```
Alternative without Redis (using SQLite):
Alternative without Redis (using SQLite):
```sh
# Set environment variables to use SQLite instead of Redis
@@ -131,19 +134,19 @@ The project consists of two main Django apps:
celery -A dashboard_project beat --scheduler django_celery_beat.schedulers:DatabaseScheduler
```
9. Run the development server:
9. Run the development server:
```sh
python manage.py runserver
```
10. Access the application at <http://127.0.0.1:8000/>
10. Access the application at `http://127.0.0.1:8000/`
### Development Workflow with UV
UV offers several advantages over traditional pip, including faster dependency resolution and installation:
1. Running linting and formatting:
1. Running linting and formatting:
```sh
# Using the convenience script
@@ -155,7 +158,7 @@ UV offers several advantages over traditional pip, including faster dependency r
uv run -m black dashboard_project
```
2. Running tests:
2. Running tests:
```sh
# Using the convenience script
@@ -165,7 +168,7 @@ UV offers several advantages over traditional pip, including faster dependency r
uv run -m pytest
```
3. Adding new dependencies:
3. Adding new dependencies:
```sh
# Add to project
@@ -176,7 +179,7 @@ UV offers several advantages over traditional pip, including faster dependency r
uv pip freeze > requirements.lock
```
4. Updating the lockfile:
4. Updating the lockfile:
```sh
uv pip compile pyproject.toml -o uv.lock
@@ -184,40 +187,41 @@ UV offers several advantages over traditional pip, including faster dependency r
### Using Docker
1. Clone the repository:
1. Clone the repository:
```sh
git clone <repository-url>
cd dashboard_project
git clone https://github.com/kjanat/livegraphs-django.git
cd livegraphs-django
```
2. Build and run with Docker Compose:
2. Build and run with Docker Compose:
```sh
docker-compose up -d --build
```
3. Create a superuser:
3. Create a superuser:
```sh
docker-compose exec web python manage.py createsuperuser
```
```sh
docker-compose exec web python manage.py createsuperuser
```
4. Access the application at <http://localhost/>
4. Access the application at <http://localhost/>
## Development Tools
### Prettier for Django Templates
This project uses Prettier with the `prettier-plugin-django-annotations` plugin to format HTML templates with Django template syntax.
This project uses Prettier with the `prettier-plugin-django-annotations` plugin to format HTML templates with Django
template syntax.
#### Prettier Configuration
The project is already configured with Prettier integration in pre-commit hooks. The configuration includes:
1. `.prettierrc` - Configuration file with Django HTML support
2. `.prettierignore` - Files to exclude from formatting
3. Pre-commit hook for automatic formatting on commits
1. `.prettierrc` - Configuration file with Django HTML support
2. `.prettierignore` - Files to exclude from formatting
3. Pre-commit hook for automatic formatting on commits
#### Manual Installation
@@ -300,18 +304,18 @@ If you need to prevent Prettier from formatting a section of your template:
The `prettier-plugin-django-annotations` plugin provides special handling for Django templates, including:
- Proper formatting of Django template tags (`{% %}`)
- Support for Django template comments (`{# #}`)
- Preservation of Django template variable output (`{{ }}`)
- Special handling for Django template syntax inside HTML attributes
- Proper formatting of Django template tags (`{% %}`)
- Support for Django template comments (`{# #}`)
- Preservation of Django template variable output (`{{ }}`)
- Special handling for Django template syntax inside HTML attributes
## Basic Usage Instructions
1. Login as the superuser you created.
2. Go to the admin interface (<http://localhost/admin/>) and create companies and users.
3. Assign users to companies.
4. Upload CSV files for each company.
5. View the analytics dashboard.
1. Login as the superuser you created.
2. Go to the admin interface (<http://localhost/admin/>) and create companies and users.
3. Assign users to companies.
4. Upload CSV files for each company.
5. View the analytics dashboard.
## Quick Start Guide
@@ -325,83 +329,86 @@ python manage.py create_sample_data
This will create:
- Admin user (username: admin, password: admin123)
- Three companies with users
- Sample chat data and dashboards
- Admin user (username: admin, password: admin123)
- Three companies with users
- Sample chat data and dashboards
### Admin Tasks
1. **Access Admin Panel**:
1. **Access Admin Panel**:
- Go to <http://localhost/admin/>
- Login with your admin credentials
- Go to <http://localhost/admin/>
- Login with your admin credentials
2. **Create a Company**:
2. **Create a Company**:
- Go to Companies > Add Company
- Fill in the company details and save
- Go to Companies > Add Company
- Fill in the company details and save
3. **Create Users**:
- Go to Users > Add User
- Fill in user details
- Assign the user to a company
- Set appropriate permissions (staff status, company admin)
3. **Create Users**:
- Go to Users > Add User
- Fill in user details
- Assign the user to a company
- Set appropriate permissions (staff status, company admin)
### Company Admin Tasks
1. **Login to Dashboard**:
1. **Login to Dashboard**:
- Go to <http://localhost/>
- Login with your company admin credentials
- Go to <http://localhost/>
- Login with your company admin credentials
2. **Upload Chat Data**:
2. **Upload Chat Data**:
- Click on "Upload Data" in the sidebar
- Fill in the data source details
- Select a CSV file containing chat data
- Click "Upload"
- Click on "Upload Data" in the sidebar
- Fill in the data source details
- Select a CSV file containing chat data
- Click "Upload"
3. **Create a Dashboard**:
- Click on "New Dashboard" in the sidebar
- Fill in the dashboard details
- Select data sources to include
- Click "Create Dashboard"
3. **Create a Dashboard**:
- Click on "New Dashboard" in the sidebar
- Fill in the dashboard details
- Select data sources to include
- Click "Create Dashboard"
### Regular User Tasks
1. **View Dashboard**:
1. **View Dashboard**:
- Login with your user credentials
- The dashboard will show automatically
- Select different dashboards from the sidebar
- Login with your user credentials
- The dashboard will show automatically
- Select different dashboards from the sidebar
2. **Search Chat Sessions**:
2. **Search Chat Sessions**:
- Click on "Search" in the top navigation
- Enter search terms
- Use filters to refine results
- Click on "Search" in the top navigation
- Enter search terms
- Use filters to refine results
3. **View Session Details**:
- In search results, click the eye icon for a session
- View complete session information and transcript
3. **View Session Details**:
- In search results, click the eye icon for a session
- View complete session information and transcript
### Dashboard Features
The dashboard includes:
- **Sessions Over Time**: Line chart showing chat volume trends
- **Sentiment Analysis**: Pie chart of positive/negative/neutral chats
- **Top Countries**: Bar chart of user countries
- **Categories**: Distribution of chat categories
- **Sessions Over Time**: Line chart showing chat volume trends
- **Sentiment Analysis**: Pie chart of positive/negative/neutral chats
- **Top Countries**: Bar chart of user countries
- **Categories**: Distribution of chat categories
### Data Source Details
View details for each data source:
- Upload date and time
- Total sessions
- Source description
- List of all chat sessions from the source
- Upload date and time
- Total sessions
- Source description
- List of all chat sessions from the source
### Troubleshooting
@@ -409,33 +416,33 @@ View details for each data source:
If your CSV upload fails:
- Ensure all required columns are present
- Check date formats (should be YYYY-MM-DD HH:MM:SS)
- Verify boolean values (TRUE/FALSE, Yes/No, 1/0)
- Check for special characters in text fields
- Ensure all required columns are present
- Check date formats (should be YYYY-MM-DD HH:MM:SS)
- Verify boolean values (TRUE/FALSE, Yes/No, 1/0)
- Check for special characters in text fields
#### Access Issues
If you can't access certain features:
- Verify your user role (admin, company admin, or regular user)
- Ensure you're assigned to the correct company
- Check if you're trying to access another company's data
- Verify your user role (admin, company admin, or regular user)
- Ensure you're assigned to the correct company
- Check if you're trying to access another company's data
#### Empty Dashboard
If your dashboard is empty:
- Verify that data sources have been uploaded
- Check that the dashboard is configured to use those data sources
- Ensure the CSV was processed successfully
- Verify that data sources have been uploaded
- Check that the dashboard is configured to use those data sources
- Ensure the CSV was processed successfully
## CSV File Format
The CSV file should contain the following columns:
| Column | Description |
| ------------------- | ------------------------------------------------------ |
|---------------------|--------------------------------------------------------|
| `session_id` | Unique identifier for the chat session |
| `start_time` | When the session started (datetime) |
| `end_time` | When the session ended (datetime) |
@@ -464,150 +471,154 @@ acme_1,2023-05-01 10:30:00,2023-05-01 10:45:00,192.168.1.1,USA,English,10,Positi
### Core Features Implemented
1. **Multi-Tenant Architecture**:
1. **Multi-Tenant Architecture**:
- Companies have isolated data and user access
- Users belong to specific companies
- Role-based permissions (admin, company admin, regular user)
- Companies have isolated data and user access
- Users belong to specific companies
- Role-based permissions (admin, company admin, regular user)
2. **Data Management**:
2. **Data Management**:
- CSV file upload and processing
- Data source management
- Chat session records with comprehensive metadata
- CSV file upload and processing
- Data source management
- Chat session records with comprehensive metadata
3. **Dashboard Visualization**:
3. **Dashboard Visualization**:
- Interactive charts using Plotly.js
- Key metrics and KPIs
- Time-series analysis
- Geographic distribution
- Sentiment analysis
- Category distribution
- Interactive charts using Plotly.js
- Key metrics and KPIs
- Time-series analysis
- Geographic distribution
- Sentiment analysis
- Category distribution
4. **Search and Analysis**:
4. **Search and Analysis**:
- Full-text search across chat sessions
- Filtering by various attributes
- Detailed view of individual chat sessions
- Transcript viewing
- Full-text search across chat sessions
- Filtering by various attributes
- Detailed view of individual chat sessions
- Transcript viewing
5. **User Management**:
5. **User Management**:
- User registration and authentication
- Profile management
- Password change functionality
- Role assignment
- User registration and authentication
- Profile management
- Password change functionality
- Role assignment
6. **Admin Interface**:
6. **Admin Interface**:
- Company management
- User administration
- Data source oversight
- System-wide configuration
- Company management
- User administration
- Data source oversight
- System-wide configuration
7. **Responsive Design**:
- Mobile-friendly interface using Bootstrap 5
- Consistent layout and navigation
- Accessible UI components
7. **Responsive Design**:
- Mobile-friendly interface using Bootstrap 5
- Consistent layout and navigation
- Accessible UI components
### Technical Implementation
#### Backend (Django)
- **Custom User Model**: Extended for company association and roles
- **Database Models**: Structured for efficient data storage and queries
- **View Logic**: Separation of concerns with dedicated view functions
- **Form Handling**: Validated data input and file uploads
- **Data Processing**: CSV parsing and structured storage
- **Template Context**: Prepared data for frontend rendering
- **URL Routing**: Clean URL structure
- **Access Control**: Permission checks throughout
- **Custom User Model**: Extended for company association and roles
- **Database Models**: Structured for efficient data storage and queries
- **View Logic**: Separation of concerns with dedicated view functions
- **Form Handling**: Validated data input and file uploads
- **Data Processing**: CSV parsing and structured storage
- **Template Context**: Prepared data for frontend rendering
- **URL Routing**: Clean URL structure
- **Access Control**: Permission checks throughout
#### Frontend
- **Bootstrap 5**: For responsive layout and UI components
- **Plotly.js**: For interactive charts and visualizations
- **jQuery**: For AJAX functionality
- **Font Awesome**: For icons
- **Custom CSS**: For styling enhancements
- **Bootstrap 5**: For responsive layout and UI components
- **Plotly.js**: For interactive charts and visualizations
- **jQuery**: For AJAX functionality
- **Font Awesome**: For icons
- **Custom CSS**: For styling enhancements
#### Data Flow
1. **Upload Process**:
1. **Upload Process**:
- File validation
- CSV parsing
- Data normalization
- Record creation
- Association with company
- File validation
- CSV parsing
- Data normalization
- Record creation
- Association with company
2. **Dashboard Generation**:
2. **Dashboard Generation**:
- Data aggregation
- Statistical calculations
- Chart data preparation
- JSON serialization for frontend
- Data aggregation
- Statistical calculations
- Chart data preparation
- JSON serialization for frontend
3. **User Authentication**:
- Login/registration handling
- Session management
- Permission checks
- Access control based on company
3. **User Authentication**:
- Login/registration handling
- Session management
- Permission checks
- Access control based on company
#### Deployment Configuration
- **Docker**: Containerization for consistent deployment
- **Docker Compose**: Multi-container orchestration
- **Nginx**: Web server and static file serving
- **PostgreSQL**: Production-ready database
- **Gunicorn**: WSGI HTTP server
- **Docker**: Containerization for consistent deployment
- **Docker Compose**: Multi-container orchestration
- **Nginx**: Web server and static file serving
- **PostgreSQL**: Production-ready database
- **Gunicorn**: WSGI HTTP server
### Models
#### Accounts App
- **CustomUser**: Extends Django's User model with company association and role
- **Company**: Represents a company with users and data sources
- **CustomUser**: Extends Django's User model with company association and role
- **Company**: Represents a company with users and data sources
#### Dashboard App
- **DataSource**: Represents an uploaded CSV file with chat data
- **ChatSession**: Stores individual chat session data parsed from CSV
- **Dashboard**: Allows configuration of custom dashboards with selected data sources
- **DataSource**: Represents an uploaded CSV file with chat data
- **ChatSession**: Stores individual chat session data parsed from CSV
- **Dashboard**: Allows configuration of custom dashboards with selected data sources
### Usage Flow
1. **Admin Setup**:
1. **Admin Setup**:
- Admin creates companies
- Admin creates users and assigns them to companies
- Admin creates companies
- Admin creates users and assigns them to companies
2. **Company Admin**:
2. **Company Admin**:
- Uploads CSV files with chat data
- Creates and configures dashboards
- Manages company users
- Uploads CSV files with chat data
- Creates and configures dashboards
- Manages company users
3. **Regular Users**:
- View dashboards
- Search and explore chat data
- Analyze chat metrics
3. **Regular Users**:
- View dashboards
- Search and explore chat data
- Analyze chat metrics
## Future Enhancements
- API integration for real-time data
- More advanced visualizations
- Custom reports
- Export to additional formats (XML, HTML, PDF)
- Theme customization
- User access control with more granular permissions
- Direct integration with chat platforms via API
- Real-time dashboard updates using WebSockets
- Advanced analytics with machine learning
- Customizable reports and scheduling
- Enhanced visualization options
- API integration for real-time data
- More advanced visualizations
- Custom reports
- Export to additional formats (XML, HTML, PDF)
- Theme customization
- User access control with more granular permissions
- Direct integration with chat platforms via API
- Real-time dashboard updates using WebSockets
- Advanced analytics with machine learning
- Customizable reports and scheduling
- Enhanced visualization options
## License
This project is unlicensed. Usage is restricted to personal and educational purposes only. For commercial use, please contact the author.
This project is unlicensed. Usage is restricted to personal and educational purposes only. For commercial use, please
contact the author.

94
TODO.md
View File

@@ -4,74 +4,74 @@
### Responsiveness
- [ ] Fix dashboard graphs scaling/adjustment when zooming (currently requires page refresh)
- [ ] Fix dashboard graphs scaling/adjustment when zooming (currently requires page refresh)
### Theming
- [x] Add dark mode/light mode toggle
- [x] Fix dark mode implementation issues:
- [x] Make charts display properly in dark mode
- [x] Fix the footer not changing color in dark mode
- [x] Adjust the sidebar nav-link styling for dark mode
- [x] Make the navbar have a different background color from the body in dark mode
- [x] Make theme toggle automatically detect and respect the user's system preference
- [x] Fix inconsistency between system dark mode preference and manual toggle
- [x] Ensure charts properly update in both scenarios (system preference and manual toggle)
- [x] Implement smooth theme transitions
- [ ] Add Notso AI branding elements
- [ ] Implement responsive table design (reduce rows to fit screen)
- [x] Add dark mode/light mode toggle
- [x] Fix dark mode implementation issues:
- [x] Make charts display properly in dark mode
- [x] Fix the footer not changing color in dark mode
- [x] Adjust the sidebar nav-link styling for dark mode
- [x] Make the navbar have a different background color from the body in dark mode
- [x] Make theme toggle automatically detect and respect the user's system preference
- [x] Fix inconsistency between system dark mode preference and manual toggle
- [x] Ensure charts properly update in both scenarios (system preference and manual toggle)
- [x] Implement smooth theme transitions
- [ ] Add Notso AI branding elements
- [ ] Implement responsive table design (reduce rows to fit screen)
### Data Export
- [x] Implement multi-format export functionality
- [x] CSV format
- [x] Excel format
- [x] JSON format
- [ ] XML format
- [ ] HTML format
- [ ] PDF format
- [ ] Create dropdown menu for export options
- [x] Make export data section collapsible (folded by default)
- [x] Add company name, date and timestamp to exported filenames
- [ ] Update [data view](dashboard_project/templates/dashboard/partials/data_table.html) to show maximum 10 rows by default, with a "Show more" button to expand to 50 rows, or "Show all" to display all rows
- [x] Implement multi-format export functionality
- [x] CSV format
- [x] Excel format
- [x] JSON format
- [ ] XML format
- [ ] HTML format
- [ ] PDF format
- [ ] Create dropdown menu for export options
- [x] Make export data section collapsible (folded by default)
- [x] Add company name, date and timestamp to exported filenames
- [ ] Update [data view](dashboard_project/templates/dashboard/partials/data_table.html) to show maximum 10 rows by default, with a "Show more" button to expand to 50 rows, or "Show all" to display all rows
## Admin Interface Enhancements
### Company Management
- [ ] Add company logo upload functionality
- [ ] Add direct CSV download button for each company (superusers only)
- [ ] Include company name, date and timestamp in filename
- [ ] Add UI for customizing CSV column names
- [ ] Add company logo upload functionality
- [ ] Add direct CSV download button for each company (superusers only)
- [ ] Include company name, date and timestamp in filename
- [ ] Add UI for customizing CSV column names
## Data Integration
### External Data Sources
- [ ] Implement periodic data download from external API
- Source: <https://proto.notso.ai/jumbo/chats>
- Authentication: Basic Auth
- Credentials: [stored securely]
- An example of the data structure can be found in [jumbo.csv](examples/jumbo.csv)
- The file that the endpoint returns is a CSV file, but the file is not a standard CSV file. It has a different structure and format:
- The header row is missing, it is supposed to be `session_id,start_time,end_time,ip_address,country,language,messages_sent,sentiment,escalated,forwarded_hr,full_transcript,avg_response_time,tokens,tokens_eur,category,initial_msg,user_rating`
- [ ] The coupling of endpoint to the company and the authentication method should be handled in the backend and the superuser should be able to change it.
- [ ] The data should be stored in the database and the dashboard should be updated with the new data.
- [ ] The csv also contains a column with full_transcript, which is a uri to a txt file, encoded in utf-8. The txt file is a raw transcript of the chat.
- [ ] The txt file should be downloaded, parsed and stored in the database.
- An example of such txt file can be found in [132f3a8c-3ba5-4d89-ae04-cd83f1bc5272.txt](examples/132f3a8c-3ba5-4d89-ae04-cd83f1bc5272.txt)
- Note that the User and Assistant messages can be multiline and can contain html, which should be safely handled, and if safe, rendered in the frontend.
- [ ] Add scheduling options for data refresh
- [ ] Add UI button to trigger manual data refresh
- [ ] Implement periodic data download from external API
- Source: <https://proto.notso.ai/jumbo/chats>
- Authentication: Basic Auth
- Credentials: stored securely
- An example of the data structure can be found in [jumbo.csv](examples/jumbo.csv)
- The file that the endpoint returns is a CSV file, but the file is not a standard CSV file. It has a different structure and format:
- The header row is missing, it is supposed to be `session_id,start_time,end_time,ip_address,country,language,messages_sent,sentiment,escalated,forwarded_hr,full_transcript,avg_response_time,tokens,tokens_eur,category,initial_msg,user_rating`
- [ ] The coupling of endpoint to the company and the authentication method should be handled in the backend and the superuser should be able to change it.
- [ ] The data should be stored in the database and the dashboard should be updated with the new data.
- [ ] The csv also contains a column with full_transcript, which is a uri to a txt file, encoded in utf-8. The txt file is a raw transcript of the chat.
- [ ] The txt file should be downloaded, parsed and stored in the database.
- An example of such txt file can be found in [132f3a8c-3ba5-4d89-ae04-cd83f1bc5272.txt](examples/132f3a8c-3ba5-4d89-ae04-cd83f1bc5272.txt)
- Note that the User and Assistant messages can be multiline and can contain html, which should be safely handled, and if safe, rendered in the frontend.
- [ ] Add scheduling options for data refresh
- [ ] Add UI button to trigger manual data refresh
## Technical Debt
### Performance Optimization
- [ ] Profile and optimize dashboard rendering
- [ ] Implement lazy loading for dashboard elements
- [ ] Profile and optimize dashboard rendering
- [ ] Implement lazy loading for dashboard elements
### Testing
- [ ] Add unit tests for export functionality
- [ ] Add integration tests for data import process
- [ ] Add unit tests for export functionality
- [ ] Add integration tests for data import process

34
docs/CELERY_REDIS.md
View File

@@ -6,10 +6,10 @@ This document explains how to set up and use Redis and Celery for background tas
The data integration module uses Celery to handle:
- Periodic data fetching from external APIs
- Processing and storing CSV data
- Downloading and parsing transcript files
- Manual data refresh triggered by users
- Periodic data fetching from external APIs
- Processing and storing CSV data
- Downloading and parsing transcript files
- Manual data refresh triggered by users
## Installation
@@ -31,13 +31,13 @@ redis-cli ping # Should output PONG
After installation, check if Redis is properly configured:
1. Open Redis configuration file:
1. Open Redis configuration file:
```bash
sudo nano /etc/redis/redis.conf
```
2. Ensure the following settings:
2. Ensure the following settings:
```bash
# For development (localhost only)
@@ -53,7 +53,7 @@ After installation, check if Redis is properly configured:
port 6379
```
3. Restart Redis after any changes:
3. Restart Redis after any changes:
```bash
sudo systemctl restart redis-server
@@ -127,25 +127,25 @@ docker-compose up -d
Development requires multiple terminal windows:
1. **Django Development Server**:
1. **Django Development Server**:
```bash
make run
```
2. **Redis Server** (if needed):
2. **Redis Server** (if needed):
```bash
make run-redis
```
3. **Celery Worker**:
3. **Celery Worker**:
```bash
make celery
```
4. **Celery Beat** (for scheduled tasks):
4. **Celery Beat** (for scheduled tasks):
```bash
make celery-beat
@@ -163,12 +163,12 @@ make run-all
If you see connection errors:
1. Check that Redis is running: `redis-cli ping` should return `PONG`
2. Verify firewall settings are not blocking port 6379
3. Check Redis binding in `/etc/redis/redis.conf` (should be `bind 127.0.0.1` for local dev)
1. Check that Redis is running: `redis-cli ping` should return `PONG`
2. Verify firewall settings are not blocking port 6379
3. Check Redis binding in `/etc/redis/redis.conf` (should be `bind 127.0.0.1` for local dev)
### Celery Workers Not Processing Tasks
1. Ensure the worker is running with the correct app name: `celery -A dashboard_project worker`
2. Check the Celery logs for errors
3. Verify broker URL settings in both code and environment variables
1. Ensure the worker is running with the correct app name: `celery -A dashboard_project worker`
2. Check the Celery logs for errors
3. Verify broker URL settings in both code and environment variables

32
docs/TROUBLESHOOTING.md
View File

@@ -25,13 +25,13 @@ python manage.py test_redis
If this fails, check the following:
1. Redis might not be running. Start it with:
1. Redis might not be running. Start it with:
```bash
sudo systemctl start redis-server
```
2. Connection credentials may be incorrect. Check your environment variables:
2. Connection credentials may be incorrect. Check your environment variables:
```bash
echo $REDIS_URL
@@ -39,13 +39,13 @@ If this fails, check the following:
echo $CELERY_RESULT_BACKEND
```
3. Redis might be binding only to a specific interface. Check `/etc/redis/redis.conf`:
3. Redis might be binding only to a specific interface. Check `/etc/redis/redis.conf`:
```bash
grep "bind" /etc/redis/redis.conf
```
4. Firewall rules might be blocking Redis. If you're connecting remotely:
4. Firewall rules might be blocking Redis. If you're connecting remotely:
```bash
sudo ufw status # Check if firewall is enabled
@@ -56,9 +56,9 @@ If this fails, check the following:
If you see the error `zip() argument 2 is shorter than argument 1`, it means the data format doesn't match the expected headers. We've implemented a fix that:
1. Pads shorter rows with empty strings
2. Uses more flexible date format parsing
3. Provides better error handling
1. Pads shorter rows with empty strings
2. Uses more flexible date format parsing
3. Provides better error handling
After these changes, your data should be processed correctly regardless of format variations.
@@ -78,14 +78,14 @@ python manage.py test_celery
If the task isn't completing, check:
1. Look for errors in the Celery worker terminal
2. Verify broker URL settings match in both terminals:
1. Look for errors in the Celery worker terminal
2. Verify broker URL settings match in both terminals:
```bash
echo $CELERY_BROKER_URL
```
3. Check if Redis is accessible from both terminals:
3. Check if Redis is accessible from both terminals:
```bash
redis-cli ping
@@ -103,36 +103,36 @@ python manage.py celery inspect scheduled
Common issues with scheduled tasks:
1. **Celery Beat not running**: Start it with:
1. **Celery Beat not running**: Start it with:
```bash
cd dashboard_project
celery -A dashboard_project beat
```
2. **Task registered but not running**: Check worker logs for any errors
2. **Task registered but not running**: Check worker logs for any errors
3. **Wrong schedule**: Check the interval in settings.py and CELERY_BEAT_SCHEDULE
3. **Wrong schedule**: Check the interval in settings.py and CELERY_BEAT_SCHEDULE
## Data Source Configuration
If data sources aren't being processed correctly:
1. Verify active data sources exist:
1. Verify active data sources exist:
```bash
cd dashboard_project
python manage.py shell -c "from data_integration.models import ExternalDataSource; print(ExternalDataSource.objects.filter(is_active=True).count())"
```
2. Create a default data source if needed:
2. Create a default data source if needed:
```bash
cd dashboard_project
python manage.py create_default_datasource
```
3. Check source URLs and credentials in the admin interface or environment variables.
3. Check source URLs and credentials in the admin interface or environment variables.
## Manually Triggering Data Refresh

14
package.json
View File

@@ -6,7 +6,7 @@
"lint:js": "oxlint",
"lint:js:fix": "bun lint:js -- --fix",
"lint:js:strict": "oxlint --import-plugin -D correctness -W suspicious",
"lint:md": "markdownlint-cli2 \"**/*.md\"",
"lint:md": "markdownlint-cli2 \"**/*.md\" \"#node_modules\" \"#.{node_modules,trunk,grit,venv}\"",
"lint:md:fix": "bun lint:md -- --fix",
"lint:py": "uvx ruff check",
"lint:py:fix": "uvx ruff check --fix",
@@ -21,17 +21,5 @@
"prettier": "^3.6.2",
"prettier-plugin-jinja-template": "^2.1.0",
"prettier-plugin-packagejson": "^2.5.19"
},
"markdownlint-cli2": {
"config": {
"MD013": false,
"MD033": false
},
"ignores": [
".git",
".trunk",
".venv",
"node_modules"
]
}
}