7043429150
- Backend: Go API server with Gin framework - Frontend: React setup (placeholder) - ML Service: Python FastAPI skeleton - Docker Compose: Full stack configuration - Database: PostgreSQL schema with migrations - Documentation: Implementation plan and README Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
1160 lines
24 KiB
Markdown
1160 lines
24 KiB
Markdown
# VictoriaLogs Manager - Implementation Plan
|
|
|
|
## Project Overview
|
|
|
|
A comprehensive web application for managing VictoriaLogs with alerting, advanced reporting, and malicious pattern detection.
|
|
|
|
## Milestones & Phases
|
|
|
|
### Milestone 1: Foundation & Authentication (Week 1)
|
|
**Goal**: Complete project setup and implement authentication system
|
|
|
|
**Duration**: 5-7 days
|
|
|
|
**Deliverables**:
|
|
- Fully configured development environment
|
|
- Working JWT authentication
|
|
- User management system
|
|
- Role-based access control
|
|
|
|
---
|
|
|
|
### Milestone 2: VictoriaLogs Integration (Week 2)
|
|
**Goal**: Implement log querying and exploration features
|
|
|
|
**Duration**: 5-7 days
|
|
|
|
**Deliverables**:
|
|
- VictoriaLogs HTTP client
|
|
- Log query API endpoints
|
|
- Logs Explorer UI
|
|
- Live tail functionality
|
|
- Export features
|
|
|
|
---
|
|
|
|
### Milestone 3: Alert Management (Week 3)
|
|
**Goal**: Complete vmalert rule management system
|
|
|
|
**Duration**: 5-7 days
|
|
|
|
**Deliverables**:
|
|
- Alert rule CRUD operations
|
|
- YAML file generation
|
|
- vmalert integration
|
|
- Alert Manager UI
|
|
- Active alerts viewer
|
|
|
|
---
|
|
|
|
### Milestone 4: Pattern Detection (Week 4)
|
|
**Goal**: Implement regex and ML-based pattern detection
|
|
|
|
**Duration**: 7-10 days
|
|
|
|
**Deliverables**:
|
|
- Regex pattern engine
|
|
- Pre-built pattern library
|
|
- ML service implementation
|
|
- Pattern Detection UI
|
|
- ML training and detection endpoints
|
|
|
|
---
|
|
|
|
### Milestone 5: Reports & Dashboards (Week 5)
|
|
**Goal**: Build visualization and reporting system
|
|
|
|
**Duration**: 5-7 days
|
|
|
|
**Deliverables**:
|
|
- Report configuration system
|
|
- Dashboard page with widgets
|
|
- ECharts integration
|
|
- Report builder UI
|
|
- Multiple chart types
|
|
|
|
---
|
|
|
|
### Milestone 6: Polish & Production (Week 6)
|
|
**Goal**: Production readiness and deployment
|
|
|
|
**Duration**: 5-7 days
|
|
|
|
**Deliverables**:
|
|
- Security hardening
|
|
- Performance optimization
|
|
- Documentation
|
|
- Testing
|
|
- Deployment guides
|
|
|
|
---
|
|
|
|
## Detailed Task Breakdown
|
|
|
|
### Milestone 1: Foundation & Authentication
|
|
|
|
#### Issue #1: Setup React Frontend
|
|
**Priority**: High
|
|
**Estimated Time**: 2-3 hours
|
|
|
|
**Tasks**:
|
|
- Initialize Vite + React + TypeScript project
|
|
- Install dependencies (React Router, Zustand, Axios, Tailwind CSS)
|
|
- Configure Tailwind CSS
|
|
- Create basic folder structure
|
|
- Setup ESLint and Prettier
|
|
|
|
**Files to Create**:
|
|
- `frontend/package.json`
|
|
- `frontend/vite.config.ts`
|
|
- `frontend/tailwind.config.js`
|
|
- `frontend/src/main.tsx`
|
|
- `frontend/src/App.tsx`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Frontend dev server runs successfully
|
|
- [ ] Tailwind CSS is working
|
|
- [ ] Hot reload is functional
|
|
|
|
---
|
|
|
|
#### Issue #2: Implement JWT Utilities
|
|
**Priority**: High
|
|
**Estimated Time**: 2 hours
|
|
|
|
**Tasks**:
|
|
- Create JWT token generation function
|
|
- Create JWT token validation function
|
|
- Implement claims structure
|
|
- Add token expiration handling
|
|
|
|
**Files to Create**:
|
|
- `backend/internal/utils/jwt.go`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Can generate JWT tokens with user claims
|
|
- [ ] Can validate and parse JWT tokens
|
|
- [ ] Token expiration is enforced
|
|
|
|
---
|
|
|
|
#### Issue #3: Implement Authentication Handlers
|
|
**Priority**: High
|
|
**Estimated Time**: 3-4 hours
|
|
|
|
**Tasks**:
|
|
- Create login handler (username/password)
|
|
- Create refresh token handler
|
|
- Create "get current user" handler
|
|
- Implement password hashing with bcrypt
|
|
- Add input validation
|
|
|
|
**Files to Create**:
|
|
- `backend/internal/api/handlers/auth.go`
|
|
- `backend/internal/services/auth_service.go`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] POST /api/v1/auth/login returns JWT token
|
|
- [ ] POST /api/v1/auth/refresh refreshes token
|
|
- [ ] GET /api/v1/auth/me returns current user
|
|
- [ ] Passwords are securely hashed
|
|
|
|
---
|
|
|
|
#### Issue #4: Implement Authentication Middleware
|
|
**Priority**: High
|
|
**Estimated Time**: 2 hours
|
|
|
|
**Tasks**:
|
|
- Create JWT validation middleware
|
|
- Create permission check middleware
|
|
- Create role check middleware
|
|
- Add error handling
|
|
|
|
**Files to Create**:
|
|
- `backend/internal/api/middleware/auth.go`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Middleware validates JWT tokens
|
|
- [ ] Invalid tokens return 401
|
|
- [ ] Permission checks work correctly
|
|
- [ ] Role checks work correctly
|
|
|
|
---
|
|
|
|
#### Issue #5: Create User Repository
|
|
**Priority**: High
|
|
**Estimated Time**: 2-3 hours
|
|
|
|
**Tasks**:
|
|
- Implement user CRUD operations
|
|
- Create database queries
|
|
- Add error handling
|
|
- Implement user lookup functions
|
|
|
|
**Files to Create**:
|
|
- `backend/internal/repository/user_repo.go`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Can create new users
|
|
- [ ] Can retrieve users by ID/username/email
|
|
- [ ] Can update user information
|
|
- [ ] Can delete users
|
|
- [ ] Can list all users
|
|
|
|
---
|
|
|
|
#### Issue #6: Implement User Management API
|
|
**Priority**: High
|
|
**Estimated Time**: 3 hours
|
|
|
|
**Tasks**:
|
|
- Create user CRUD endpoints
|
|
- Add role-based access control
|
|
- Implement user service layer
|
|
- Add input validation
|
|
|
|
**Files to Create**:
|
|
- `backend/internal/api/handlers/users.go`
|
|
- `backend/internal/services/user_service.go`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] GET /api/v1/users returns user list (admin only)
|
|
- [ ] POST /api/v1/users creates user (admin only)
|
|
- [ ] PUT /api/v1/users/:id updates user (admin only)
|
|
- [ ] DELETE /api/v1/users/:id deletes user (admin only)
|
|
|
|
---
|
|
|
|
#### Issue #7: Create Auth Store (Frontend)
|
|
**Priority**: High
|
|
**Estimated Time**: 2 hours
|
|
|
|
**Tasks**:
|
|
- Setup Zustand store for authentication
|
|
- Implement login/logout functions
|
|
- Add token persistence (localStorage)
|
|
- Create auth hooks
|
|
|
|
**Files to Create**:
|
|
- `frontend/src/store/authStore.ts`
|
|
- `frontend/src/hooks/useAuth.ts`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Auth state persists across page reloads
|
|
- [ ] Login updates auth state
|
|
- [ ] Logout clears auth state
|
|
|
|
---
|
|
|
|
#### Issue #8: Create API Client (Frontend)
|
|
**Priority**: High
|
|
**Estimated Time**: 2 hours
|
|
|
|
**Tasks**:
|
|
- Setup Axios instance
|
|
- Add request interceptor (JWT token)
|
|
- Add response interceptor (401 handling)
|
|
- Create API client functions
|
|
|
|
**Files to Create**:
|
|
- `frontend/src/api/client.ts`
|
|
- `frontend/src/api/auth.ts`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] All requests include JWT token
|
|
- [ ] 401 responses trigger logout
|
|
- [ ] API client is properly typed
|
|
|
|
---
|
|
|
|
#### Issue #9: Create Login Page
|
|
**Priority**: High
|
|
**Estimated Time**: 3 hours
|
|
|
|
**Tasks**:
|
|
- Create login form UI
|
|
- Add form validation
|
|
- Implement login submission
|
|
- Add error handling
|
|
- Add loading states
|
|
|
|
**Files to Create**:
|
|
- `frontend/src/pages/Login.tsx`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Login form validates inputs
|
|
- [ ] Successful login redirects to dashboard
|
|
- [ ] Error messages display correctly
|
|
- [ ] Loading state shows during request
|
|
|
|
---
|
|
|
|
#### Issue #10: Create Protected Route Component
|
|
**Priority**: High
|
|
**Estimated Time**: 1 hour
|
|
|
|
**Tasks**:
|
|
- Create ProtectedRoute component
|
|
- Add authentication check
|
|
- Add role-based access
|
|
- Implement redirects
|
|
|
|
**Files to Create**:
|
|
- `frontend/src/components/common/ProtectedRoute.tsx`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Unauthenticated users redirect to login
|
|
- [ ] Unauthorized users see error message
|
|
- [ ] Authorized users see content
|
|
|
|
---
|
|
|
|
### Milestone 2: VictoriaLogs Integration
|
|
|
|
#### Issue #11: Create VictoriaLogs HTTP Client
|
|
**Priority**: High
|
|
**Estimated Time**: 4 hours
|
|
|
|
**Tasks**:
|
|
- Create VictoriaLogs HTTP client
|
|
- Implement query method
|
|
- Implement tail method
|
|
- Implement facets method
|
|
- Implement stats query method
|
|
- Add error handling
|
|
|
|
**Files to Create**:
|
|
- `backend/pkg/vlogs/client.go`
|
|
- `backend/pkg/vlogs/models.go`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Can query VictoriaLogs
|
|
- [ ] Can tail logs in real-time
|
|
- [ ] Can fetch facets
|
|
- [ ] Can execute stats queries
|
|
|
|
---
|
|
|
|
#### Issue #12: Create Log Query Service
|
|
**Priority**: High
|
|
**Estimated Time**: 3 hours
|
|
|
|
**Tasks**:
|
|
- Implement log query proxy
|
|
- Add authentication checks
|
|
- Implement rate limiting
|
|
- Add query validation
|
|
|
|
**Files to Create**:
|
|
- `backend/internal/services/vlogs_service.go`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Service proxies requests to VictoriaLogs
|
|
- [ ] User permissions are enforced
|
|
- [ ] Rate limiting works per role
|
|
|
|
---
|
|
|
|
#### Issue #13: Implement Log Query API Endpoints
|
|
**Priority**: High
|
|
**Estimated Time**: 3 hours
|
|
|
|
**Tasks**:
|
|
- Create POST /logs/query endpoint
|
|
- Create POST /logs/tail endpoint
|
|
- Create GET /logs/facets endpoint
|
|
- Create POST /logs/stats endpoint
|
|
- Create POST /logs/export endpoint
|
|
|
|
**Files to Create**:
|
|
- `backend/internal/api/handlers/logs.go`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] All endpoints return correct data
|
|
- [ ] Authentication is required
|
|
- [ ] Error handling is robust
|
|
|
|
---
|
|
|
|
#### Issue #14: Create Logs API Client (Frontend)
|
|
**Priority**: High
|
|
**Estimated Time**: 2 hours
|
|
|
|
**Tasks**:
|
|
- Create logs API functions
|
|
- Add TypeScript types
|
|
- Implement query builder helper
|
|
|
|
**Files to Create**:
|
|
- `frontend/src/api/logs.ts`
|
|
- `frontend/src/types/logs.ts`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Can query logs from frontend
|
|
- [ ] All functions are properly typed
|
|
|
|
---
|
|
|
|
#### Issue #15: Create Logs Store (Frontend)
|
|
**Priority**: Medium
|
|
**Estimated Time**: 2 hours
|
|
|
|
**Tasks**:
|
|
- Create Zustand store for logs
|
|
- Implement query state management
|
|
- Add pagination state
|
|
- Add filter state
|
|
|
|
**Files to Create**:
|
|
- `frontend/src/store/logsStore.ts`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Store manages query state
|
|
- [ ] Pagination works correctly
|
|
- [ ] Filters are applied properly
|
|
|
|
---
|
|
|
|
#### Issue #16: Create Query Builder Component
|
|
**Priority**: High
|
|
**Estimated Time**: 4-5 hours
|
|
|
|
**Tasks**:
|
|
- Create visual query builder UI
|
|
- Add field selector
|
|
- Add operator selector
|
|
- Add value input
|
|
- Implement query preview
|
|
- Add raw LogsQL editor toggle
|
|
|
|
**Files to Create**:
|
|
- `frontend/src/components/logs/QueryBuilder.tsx`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Can build queries visually
|
|
- [ ] Can switch to raw LogsQL
|
|
- [ ] Query preview is accurate
|
|
|
|
---
|
|
|
|
#### Issue #17: Create Log Table Component
|
|
**Priority**: High
|
|
**Estimated Time**: 4 hours
|
|
|
|
**Tasks**:
|
|
- Create virtualized table (react-window)
|
|
- Implement expandable rows
|
|
- Add column customization
|
|
- Add sorting
|
|
- Add loading states
|
|
|
|
**Files to Create**:
|
|
- `frontend/src/components/logs/LogTable.tsx`
|
|
- `frontend/src/components/logs/LogDetails.tsx`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Table performs well with 10,000+ rows
|
|
- [ ] Rows expand to show details
|
|
- [ ] Columns are customizable
|
|
|
|
---
|
|
|
|
#### Issue #18: Create Logs Explorer Page
|
|
**Priority**: High
|
|
**Estimated Time**: 4 hours
|
|
|
|
**Tasks**:
|
|
- Create main Logs Explorer page
|
|
- Integrate QueryBuilder
|
|
- Integrate LogTable
|
|
- Add time range selector
|
|
- Add facets sidebar
|
|
- Add export button
|
|
|
|
**Files to Create**:
|
|
- `frontend/src/pages/LogsExplorer.tsx`
|
|
- `frontend/src/components/logs/TimeRangeSelector.tsx`
|
|
- `frontend/src/components/logs/FacetsSidebar.tsx`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Can query and view logs
|
|
- [ ] Time range selector works
|
|
- [ ] Facets are clickable filters
|
|
|
|
---
|
|
|
|
#### Issue #19: Implement Live Tail (WebSocket)
|
|
**Priority**: Medium
|
|
**Estimated Time**: 4 hours
|
|
|
|
**Tasks**:
|
|
- Implement WebSocket endpoint (backend)
|
|
- Create live tail component (frontend)
|
|
- Add auto-scroll
|
|
- Add pause/resume
|
|
- Add disconnect handling
|
|
|
|
**Files to Create**:
|
|
- `backend/internal/api/handlers/websocket.go`
|
|
- `frontend/src/components/logs/LiveTail.tsx`
|
|
- `frontend/src/hooks/useWebSocket.ts`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Logs stream in real-time
|
|
- [ ] Can pause/resume streaming
|
|
- [ ] Handles disconnections gracefully
|
|
|
|
---
|
|
|
|
#### Issue #20: Implement Log Export
|
|
**Priority**: Low
|
|
**Estimated Time**: 2 hours
|
|
|
|
**Tasks**:
|
|
- Add CSV export
|
|
- Add JSON export
|
|
- Add download handling
|
|
|
|
**Files to Modify**:
|
|
- `backend/internal/api/handlers/logs.go`
|
|
- `frontend/src/components/logs/ExportButton.tsx`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Can export logs as CSV
|
|
- [ ] Can export logs as JSON
|
|
- [ ] Large exports stream properly
|
|
|
|
---
|
|
|
|
### Milestone 3: Alert Management
|
|
|
|
#### Issue #21: Create Alert Rule Models
|
|
**Priority**: High
|
|
**Estimated Time**: 1 hour
|
|
|
|
**Tasks**:
|
|
- Define alert rule struct
|
|
- Add JSON/YAML tags
|
|
- Create validation functions
|
|
|
|
**Files to Create**:
|
|
- `backend/internal/models/alert.go`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Alert rule model matches schema
|
|
- [ ] Validation functions work
|
|
|
|
---
|
|
|
|
#### Issue #22: Create Alert Repository
|
|
**Priority**: High
|
|
**Estimated Time**: 3 hours
|
|
|
|
**Tasks**:
|
|
- Implement alert rule CRUD
|
|
- Add database queries
|
|
- Implement rule search/filter
|
|
|
|
**Files to Create**:
|
|
- `backend/internal/repository/alert_repo.go`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Can create/read/update/delete alert rules
|
|
- [ ] Can filter rules by group/status
|
|
|
|
---
|
|
|
|
#### Issue #23: Implement vmalert Service
|
|
**Priority**: High
|
|
**Estimated Time**: 5 hours
|
|
|
|
**Tasks**:
|
|
- Create vmalert service
|
|
- Implement YAML file generation
|
|
- Implement vmalert reload trigger
|
|
- Add active alerts fetching
|
|
- Add validation
|
|
|
|
**Files to Create**:
|
|
- `backend/internal/services/vmalert_service.go`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Can generate valid YAML files
|
|
- [ ] vmalert reloads successfully
|
|
- [ ] Can fetch active alerts from vmalert
|
|
|
|
---
|
|
|
|
#### Issue #24: Implement Alert API Endpoints
|
|
**Priority**: High
|
|
**Estimated Time**: 3 hours
|
|
|
|
**Tasks**:
|
|
- Create alert rule CRUD endpoints
|
|
- Add reload endpoint
|
|
- Add active alerts endpoint
|
|
- Add permissions checks
|
|
|
|
**Files to Create**:
|
|
- `backend/internal/api/handlers/alerts.go`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] All CRUD operations work
|
|
- [ ] Reload triggers vmalert reload
|
|
- [ ] Active alerts are fetched
|
|
|
|
---
|
|
|
|
#### Issue #25: Create Alert API Client (Frontend)
|
|
**Priority**: High
|
|
**Estimated Time**: 2 hours
|
|
|
|
**Tasks**:
|
|
- Create alerts API functions
|
|
- Add TypeScript types
|
|
|
|
**Files to Create**:
|
|
- `frontend/src/api/alerts.ts`
|
|
- `frontend/src/types/alerts.ts`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Can manage alerts from frontend
|
|
- [ ] All functions are typed
|
|
|
|
---
|
|
|
|
#### Issue #26: Create Alert Rule Form Component
|
|
**Priority**: High
|
|
**Estimated Time**: 4 hours
|
|
|
|
**Tasks**:
|
|
- Create alert rule form
|
|
- Add field validation
|
|
- Add LogsQL query builder
|
|
- Add labels/annotations editor
|
|
|
|
**Files to Create**:
|
|
- `frontend/src/components/alerts/AlertRuleForm.tsx`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Form validates all fields
|
|
- [ ] Can create/edit rules
|
|
- [ ] LogsQL syntax is validated
|
|
|
|
---
|
|
|
|
#### Issue #27: Create Alert Rule YAML Editor
|
|
**Priority**: Medium
|
|
**Estimated Time**: 3 hours
|
|
|
|
**Tasks**:
|
|
- Integrate Monaco Editor
|
|
- Add YAML syntax highlighting
|
|
- Add YAML validation
|
|
- Add save functionality
|
|
|
|
**Files to Create**:
|
|
- `frontend/src/components/alerts/AlertRuleEditor.tsx`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] YAML editor has syntax highlighting
|
|
- [ ] Invalid YAML shows errors
|
|
- [ ] Can save YAML directly
|
|
|
|
---
|
|
|
|
#### Issue #28: Create Alerts Manager Page
|
|
**Priority**: High
|
|
**Estimated Time**: 4 hours
|
|
|
|
**Tasks**:
|
|
- Create alerts list view
|
|
- Add create/edit/delete actions
|
|
- Add active alerts section
|
|
- Add alert history
|
|
|
|
**Files to Create**:
|
|
- `frontend/src/pages/AlertsManager.tsx`
|
|
- `frontend/src/components/alerts/AlertRuleList.tsx`
|
|
- `frontend/src/components/alerts/AlertStatus.tsx`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Can view all alert rules
|
|
- [ ] Can create/edit/delete rules
|
|
- [ ] Active alerts are displayed
|
|
|
|
---
|
|
|
|
### Milestone 4: Pattern Detection
|
|
|
|
#### Issue #29: Create Pattern Models
|
|
**Priority**: High
|
|
**Estimated Time**: 1 hour
|
|
|
|
**Tasks**:
|
|
- Define pattern struct
|
|
- Add validation
|
|
- Create pre-built patterns
|
|
|
|
**Files to Create**:
|
|
- `backend/internal/models/pattern.go`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Pattern model is complete
|
|
- [ ] Pre-built patterns are defined
|
|
|
|
---
|
|
|
|
#### Issue #30: Create Pattern Repository
|
|
**Priority**: High
|
|
**Estimated Time**: 2 hours
|
|
|
|
**Tasks**:
|
|
- Implement pattern CRUD
|
|
- Add pattern search
|
|
|
|
**Files to Create**:
|
|
- `backend/internal/repository/pattern_repo.go`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Can manage patterns in database
|
|
|
|
---
|
|
|
|
#### Issue #31: Implement Regex Pattern Detection
|
|
**Priority**: High
|
|
**Estimated Time**: 4 hours
|
|
|
|
**Tasks**:
|
|
- Create pattern detection engine
|
|
- Implement regex matching
|
|
- Generate LogsQL queries
|
|
- Execute detection
|
|
|
|
**Files to Create**:
|
|
- `backend/internal/services/pattern_service.go`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Can detect patterns using regex
|
|
- [ ] Returns matched logs
|
|
|
|
---
|
|
|
|
#### Issue #32: Implement ML Feature Extraction
|
|
**Priority**: High
|
|
**Estimated Time**: 4 hours
|
|
|
|
**Tasks**:
|
|
- Create feature extractor
|
|
- Extract temporal features
|
|
- Extract log content features
|
|
- Extract numerical features
|
|
|
|
**Files to Create**:
|
|
- `ml-service/app/utils/feature_extraction.py`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Extracts relevant features from logs
|
|
- [ ] Features are properly normalized
|
|
|
|
---
|
|
|
|
#### Issue #33: Implement ML Anomaly Detector
|
|
**Priority**: High
|
|
**Estimated Time**: 5 hours
|
|
|
|
**Tasks**:
|
|
- Implement Isolation Forest model
|
|
- Add training functionality
|
|
- Add prediction functionality
|
|
- Add model persistence
|
|
|
|
**Files to Create**:
|
|
- `ml-service/app/models/anomaly_detector.py`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Model trains successfully
|
|
- [ ] Model detects anomalies
|
|
- [ ] Model can be saved/loaded
|
|
|
|
---
|
|
|
|
#### Issue #34: Implement ML Service Endpoints
|
|
**Priority**: High
|
|
**Estimated Time**: 3 hours
|
|
|
|
**Tasks**:
|
|
- Complete /api/ml/train endpoint
|
|
- Complete /api/ml/detect endpoint
|
|
- Add VictoriaLogs integration
|
|
|
|
**Files to Modify**:
|
|
- `ml-service/app/main.py`
|
|
- `ml-service/app/api/endpoints.py`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Training endpoint works
|
|
- [ ] Detection endpoint works
|
|
- [ ] Fetches logs from VictoriaLogs
|
|
|
|
---
|
|
|
|
#### Issue #35: Create ML Service Client (Backend)
|
|
**Priority**: High
|
|
**Estimated Time**: 2 hours
|
|
|
|
**Tasks**:
|
|
- Create ML service HTTP client
|
|
- Implement train/detect methods
|
|
|
|
**Files to Create**:
|
|
- `backend/internal/services/ml_service.go`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Can communicate with ML service
|
|
- [ ] Handles timeouts properly
|
|
|
|
---
|
|
|
|
#### Issue #36: Create Pattern Detection Page
|
|
**Priority**: High
|
|
**Estimated Time**: 5 hours
|
|
|
|
**Tasks**:
|
|
- Create pattern library UI
|
|
- Add regex pattern form
|
|
- Add ML detection trigger
|
|
- Display detection results
|
|
|
|
**Files to Create**:
|
|
- `frontend/src/pages/PatternDetection.tsx`
|
|
- `frontend/src/components/patterns/PatternLibrary.tsx`
|
|
- `frontend/src/components/patterns/RegexPatternForm.tsx`
|
|
- `frontend/src/components/patterns/MLAnomalyViewer.tsx`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Can browse pattern library
|
|
- [ ] Can create custom patterns
|
|
- [ ] Can trigger ML detection
|
|
- [ ] Results are displayed clearly
|
|
|
|
---
|
|
|
|
### Milestone 5: Reports & Dashboards
|
|
|
|
#### Issue #37: Create Report Models
|
|
**Priority**: High
|
|
**Estimated Time**: 1 hour
|
|
|
|
**Tasks**:
|
|
- Define report struct
|
|
- Define widget configs
|
|
|
|
**Files to Create**:
|
|
- `backend/internal/models/report.go`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Report model is complete
|
|
|
|
---
|
|
|
|
#### Issue #38: Create Report Repository
|
|
**Priority**: High
|
|
**Estimated Time**: 2 hours
|
|
|
|
**Tasks**:
|
|
- Implement report CRUD
|
|
|
|
**Files to Create**:
|
|
- `backend/internal/repository/report_repo.go`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Can manage reports
|
|
|
|
---
|
|
|
|
#### Issue #39: Implement Report Execution Engine
|
|
**Priority**: High
|
|
**Estimated Time**: 4 hours
|
|
|
|
**Tasks**:
|
|
- Create report execution service
|
|
- Execute queries for widgets
|
|
- Aggregate data
|
|
|
|
**Files to Create**:
|
|
- `backend/internal/services/report_service.go`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Can execute report queries
|
|
- [ ] Returns formatted data for charts
|
|
|
|
---
|
|
|
|
#### Issue #40: Integrate ECharts
|
|
**Priority**: High
|
|
**Estimated Time**: 2 hours
|
|
|
|
**Tasks**:
|
|
- Install echarts-for-react
|
|
- Create chart wrapper components
|
|
|
|
**Files to Create**:
|
|
- `frontend/src/components/charts/BaseChart.tsx`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] ECharts renders correctly
|
|
- [ ] Charts are responsive
|
|
|
|
---
|
|
|
|
#### Issue #41: Create Chart Components
|
|
**Priority**: High
|
|
**Estimated Time**: 4 hours
|
|
|
|
**Tasks**:
|
|
- Create TimeSeriesChart
|
|
- Create PieChart
|
|
- Create BarChart
|
|
- Create HeatmapChart
|
|
|
|
**Files to Create**:
|
|
- `frontend/src/components/charts/TimeSeriesChart.tsx`
|
|
- `frontend/src/components/charts/PieChart.tsx`
|
|
- `frontend/src/components/charts/BarChart.tsx`
|
|
- `frontend/src/components/charts/HeatmapChart.tsx`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] All chart types render correctly
|
|
- [ ] Charts are interactive
|
|
|
|
---
|
|
|
|
#### Issue #42: Create Dashboard Page
|
|
**Priority**: High
|
|
**Estimated Time**: 5 hours
|
|
|
|
**Tasks**:
|
|
- Create dashboard layout
|
|
- Add summary cards
|
|
- Add multiple chart widgets
|
|
- Make layout responsive
|
|
|
|
**Files to Create**:
|
|
- `frontend/src/pages/Dashboard.tsx`
|
|
- `frontend/src/components/dashboard/SummaryCard.tsx`
|
|
- `frontend/src/components/dashboard/DashboardGrid.tsx`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Dashboard displays key metrics
|
|
- [ ] Multiple widgets are shown
|
|
- [ ] Layout is responsive
|
|
|
|
---
|
|
|
|
#### Issue #43: Create Report Builder UI
|
|
**Priority**: Medium
|
|
**Estimated Time**: 5 hours
|
|
|
|
**Tasks**:
|
|
- Create report builder interface
|
|
- Add widget configuration
|
|
- Add query builder for widgets
|
|
- Add save/load functionality
|
|
|
|
**Files to Create**:
|
|
- `frontend/src/components/reports/ReportBuilder.tsx`
|
|
- `frontend/src/components/reports/WidgetConfig.tsx`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Can create custom reports
|
|
- [ ] Can configure widgets
|
|
- [ ] Reports are saved
|
|
|
|
---
|
|
|
|
### Milestone 6: Polish & Production
|
|
|
|
#### Issue #44: Implement Rate Limiting
|
|
**Priority**: High
|
|
**Estimated Time**: 3 hours
|
|
|
|
**Tasks**:
|
|
- Add rate limiting middleware
|
|
- Configure limits per role
|
|
- Add rate limit headers
|
|
|
|
**Files to Create**:
|
|
- `backend/internal/api/middleware/ratelimit.go`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Rate limiting works per role
|
|
- [ ] Returns 429 when exceeded
|
|
|
|
---
|
|
|
|
#### Issue #45: Add Input Validation
|
|
**Priority**: High
|
|
**Estimated Time**: 3 hours
|
|
|
|
**Tasks**:
|
|
- Add query validation
|
|
- Add pattern validation
|
|
- Add length limits
|
|
- Add SQL injection prevention
|
|
|
|
**Files to Create**:
|
|
- `backend/internal/utils/validator.go`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Dangerous inputs are rejected
|
|
- [ ] Validation messages are clear
|
|
|
|
---
|
|
|
|
#### Issue #46: Add Audit Logging
|
|
**Priority**: High
|
|
**Estimated Time**: 3 hours
|
|
|
|
**Tasks**:
|
|
- Create audit logging middleware
|
|
- Log sensitive operations
|
|
- Add audit log viewer (admin)
|
|
|
|
**Files to Create**:
|
|
- `backend/internal/api/middleware/audit.go`
|
|
- `backend/internal/services/audit_service.go`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] All sensitive operations are logged
|
|
- [ ] Admins can view audit logs
|
|
|
|
---
|
|
|
|
#### Issue #47: Add Loading States and Error Handling
|
|
**Priority**: Medium
|
|
**Estimated Time**: 4 hours
|
|
|
|
**Tasks**:
|
|
- Add loading spinners
|
|
- Add error toast notifications
|
|
- Improve error messages
|
|
- Add retry logic
|
|
|
|
**Files to Create**:
|
|
- `frontend/src/components/common/LoadingSpinner.tsx`
|
|
- `frontend/src/components/common/Toast.tsx`
|
|
- `frontend/src/hooks/useToast.ts`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Loading states show during requests
|
|
- [ ] Errors display user-friendly messages
|
|
|
|
---
|
|
|
|
#### Issue #48: Optimize Performance
|
|
**Priority**: Medium
|
|
**Estimated Time**: 4 hours
|
|
|
|
**Tasks**:
|
|
- Add Redis caching (optional)
|
|
- Optimize database queries
|
|
- Add frontend code splitting
|
|
- Optimize bundle size
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Page load time < 2s
|
|
- [ ] Bundle size < 500KB
|
|
|
|
---
|
|
|
|
#### Issue #49: Write API Documentation
|
|
**Priority**: Medium
|
|
**Estimated Time**: 4 hours
|
|
|
|
**Tasks**:
|
|
- Document all API endpoints
|
|
- Add request/response examples
|
|
- Create Postman collection
|
|
|
|
**Files to Create**:
|
|
- `docs/api.md`
|
|
- `docs/postman_collection.json`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] All endpoints are documented
|
|
- [ ] Examples are provided
|
|
|
|
---
|
|
|
|
#### Issue #50: Write User Guide
|
|
**Priority**: Low
|
|
**Estimated Time**: 3 hours
|
|
|
|
**Tasks**:
|
|
- Write getting started guide
|
|
- Document features
|
|
- Add screenshots
|
|
|
|
**Files to Create**:
|
|
- `docs/user-guide.md`
|
|
- `docs/screenshots/`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] User guide is complete
|
|
- [ ] Screenshots are included
|
|
|
|
---
|
|
|
|
#### Issue #51: Write Deployment Guide
|
|
**Priority**: Medium
|
|
**Estimated Time**: 3 hours
|
|
|
|
**Tasks**:
|
|
- Document Docker deployment
|
|
- Document production configuration
|
|
- Add troubleshooting section
|
|
|
|
**Files to Create**:
|
|
- `docs/deployment.md`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Deployment guide is complete
|
|
- [ ] Production checklist is provided
|
|
|
|
---
|
|
|
|
#### Issue #52: Add Unit Tests
|
|
**Priority**: Medium
|
|
**Estimated Time**: 6 hours
|
|
|
|
**Tasks**:
|
|
- Write backend unit tests (critical paths)
|
|
- Write frontend unit tests (critical components)
|
|
- Add test CI pipeline
|
|
|
|
**Files to Create**:
|
|
- `backend/*_test.go`
|
|
- `frontend/src/**/*.test.tsx`
|
|
- `.github/workflows/test.yml` or `.gitea/workflows/test.yml`
|
|
|
|
**Acceptance Criteria**:
|
|
- [ ] Critical paths have tests
|
|
- [ ] Tests pass in CI
|
|
|
|
---
|
|
|
|
## Summary
|
|
|
|
**Total Estimated Time**: 160-200 hours (4-5 weeks)
|
|
|
|
**Total Issues**: 52
|
|
|
|
**Total Milestones**: 6
|
|
|
|
**Priority Breakdown**:
|
|
- High Priority: 38 issues
|
|
- Medium Priority: 11 issues
|
|
- Low Priority: 3 issues
|