# VictoriaLogs Manager - Implementation Plan ## Project Overview A comprehensive web application for managing VictoriaLogs with alerting, advanced reporting, and malicious pattern detection. ## Milestones & Phases ### Milestone 1: Foundation & Authentication (Week 1) **Goal**: Complete project setup and implement authentication system **Duration**: 5-7 days **Deliverables**: - Fully configured development environment - Working JWT authentication - User management system - Role-based access control --- ### Milestone 2: VictoriaLogs Integration (Week 2) **Goal**: Implement log querying and exploration features **Duration**: 5-7 days **Deliverables**: - VictoriaLogs HTTP client - Log query API endpoints - Logs Explorer UI - Live tail functionality - Export features --- ### Milestone 3: Alert Management (Week 3) **Goal**: Complete vmalert rule management system **Duration**: 5-7 days **Deliverables**: - Alert rule CRUD operations - YAML file generation - vmalert integration - Alert Manager UI - Active alerts viewer --- ### Milestone 4: Pattern Detection (Week 4) **Goal**: Implement regex and ML-based pattern detection **Duration**: 7-10 days **Deliverables**: - Regex pattern engine - Pre-built pattern library - ML service implementation - Pattern Detection UI - ML training and detection endpoints --- ### Milestone 5: Reports & Dashboards (Week 5) **Goal**: Build visualization and reporting system **Duration**: 5-7 days **Deliverables**: - Report configuration system - Dashboard page with widgets - ECharts integration - Report builder UI - Multiple chart types --- ### Milestone 6: Polish & Production (Week 6) **Goal**: Production readiness and deployment **Duration**: 5-7 days **Deliverables**: - Security hardening - Performance optimization - Documentation - Testing - Deployment guides --- ## Detailed Task Breakdown ### Milestone 1: Foundation & Authentication #### Issue #1: Setup React Frontend **Priority**: High **Estimated Time**: 2-3 hours **Tasks**: - Initialize Vite + React + TypeScript project - Install dependencies (React Router, Zustand, Axios, Tailwind CSS) - Configure Tailwind CSS - Create basic folder structure - Setup ESLint and Prettier **Files to Create**: - `frontend/package.json` - `frontend/vite.config.ts` - `frontend/tailwind.config.js` - `frontend/src/main.tsx` - `frontend/src/App.tsx` **Acceptance Criteria**: - [ ] Frontend dev server runs successfully - [ ] Tailwind CSS is working - [ ] Hot reload is functional --- #### Issue #2: Implement JWT Utilities **Priority**: High **Estimated Time**: 2 hours **Tasks**: - Create JWT token generation function - Create JWT token validation function - Implement claims structure - Add token expiration handling **Files to Create**: - `backend/internal/utils/jwt.go` **Acceptance Criteria**: - [ ] Can generate JWT tokens with user claims - [ ] Can validate and parse JWT tokens - [ ] Token expiration is enforced --- #### Issue #3: Implement Authentication Handlers **Priority**: High **Estimated Time**: 3-4 hours **Tasks**: - Create login handler (username/password) - Create refresh token handler - Create "get current user" handler - Implement password hashing with bcrypt - Add input validation **Files to Create**: - `backend/internal/api/handlers/auth.go` - `backend/internal/services/auth_service.go` **Acceptance Criteria**: - [ ] POST /api/v1/auth/login returns JWT token - [ ] POST /api/v1/auth/refresh refreshes token - [ ] GET /api/v1/auth/me returns current user - [ ] Passwords are securely hashed --- #### Issue #4: Implement Authentication Middleware **Priority**: High **Estimated Time**: 2 hours **Tasks**: - Create JWT validation middleware - Create permission check middleware - Create role check middleware - Add error handling **Files to Create**: - `backend/internal/api/middleware/auth.go` **Acceptance Criteria**: - [ ] Middleware validates JWT tokens - [ ] Invalid tokens return 401 - [ ] Permission checks work correctly - [ ] Role checks work correctly --- #### Issue #5: Create User Repository **Priority**: High **Estimated Time**: 2-3 hours **Tasks**: - Implement user CRUD operations - Create database queries - Add error handling - Implement user lookup functions **Files to Create**: - `backend/internal/repository/user_repo.go` **Acceptance Criteria**: - [ ] Can create new users - [ ] Can retrieve users by ID/username/email - [ ] Can update user information - [ ] Can delete users - [ ] Can list all users --- #### Issue #6: Implement User Management API **Priority**: High **Estimated Time**: 3 hours **Tasks**: - Create user CRUD endpoints - Add role-based access control - Implement user service layer - Add input validation **Files to Create**: - `backend/internal/api/handlers/users.go` - `backend/internal/services/user_service.go` **Acceptance Criteria**: - [ ] GET /api/v1/users returns user list (admin only) - [ ] POST /api/v1/users creates user (admin only) - [ ] PUT /api/v1/users/:id updates user (admin only) - [ ] DELETE /api/v1/users/:id deletes user (admin only) --- #### Issue #7: Create Auth Store (Frontend) **Priority**: High **Estimated Time**: 2 hours **Tasks**: - Setup Zustand store for authentication - Implement login/logout functions - Add token persistence (localStorage) - Create auth hooks **Files to Create**: - `frontend/src/store/authStore.ts` - `frontend/src/hooks/useAuth.ts` **Acceptance Criteria**: - [ ] Auth state persists across page reloads - [ ] Login updates auth state - [ ] Logout clears auth state --- #### Issue #8: Create API Client (Frontend) **Priority**: High **Estimated Time**: 2 hours **Tasks**: - Setup Axios instance - Add request interceptor (JWT token) - Add response interceptor (401 handling) - Create API client functions **Files to Create**: - `frontend/src/api/client.ts` - `frontend/src/api/auth.ts` **Acceptance Criteria**: - [ ] All requests include JWT token - [ ] 401 responses trigger logout - [ ] API client is properly typed --- #### Issue #9: Create Login Page **Priority**: High **Estimated Time**: 3 hours **Tasks**: - Create login form UI - Add form validation - Implement login submission - Add error handling - Add loading states **Files to Create**: - `frontend/src/pages/Login.tsx` **Acceptance Criteria**: - [ ] Login form validates inputs - [ ] Successful login redirects to dashboard - [ ] Error messages display correctly - [ ] Loading state shows during request --- #### Issue #10: Create Protected Route Component **Priority**: High **Estimated Time**: 1 hour **Tasks**: - Create ProtectedRoute component - Add authentication check - Add role-based access - Implement redirects **Files to Create**: - `frontend/src/components/common/ProtectedRoute.tsx` **Acceptance Criteria**: - [ ] Unauthenticated users redirect to login - [ ] Unauthorized users see error message - [ ] Authorized users see content --- ### Milestone 2: VictoriaLogs Integration #### Issue #11: Create VictoriaLogs HTTP Client **Priority**: High **Estimated Time**: 4 hours **Tasks**: - Create VictoriaLogs HTTP client - Implement query method - Implement tail method - Implement facets method - Implement stats query method - Add error handling **Files to Create**: - `backend/pkg/vlogs/client.go` - `backend/pkg/vlogs/models.go` **Acceptance Criteria**: - [ ] Can query VictoriaLogs - [ ] Can tail logs in real-time - [ ] Can fetch facets - [ ] Can execute stats queries --- #### Issue #12: Create Log Query Service **Priority**: High **Estimated Time**: 3 hours **Tasks**: - Implement log query proxy - Add authentication checks - Implement rate limiting - Add query validation **Files to Create**: - `backend/internal/services/vlogs_service.go` **Acceptance Criteria**: - [ ] Service proxies requests to VictoriaLogs - [ ] User permissions are enforced - [ ] Rate limiting works per role --- #### Issue #13: Implement Log Query API Endpoints **Priority**: High **Estimated Time**: 3 hours **Tasks**: - Create POST /logs/query endpoint - Create POST /logs/tail endpoint - Create GET /logs/facets endpoint - Create POST /logs/stats endpoint - Create POST /logs/export endpoint **Files to Create**: - `backend/internal/api/handlers/logs.go` **Acceptance Criteria**: - [ ] All endpoints return correct data - [ ] Authentication is required - [ ] Error handling is robust --- #### Issue #14: Create Logs API Client (Frontend) **Priority**: High **Estimated Time**: 2 hours **Tasks**: - Create logs API functions - Add TypeScript types - Implement query builder helper **Files to Create**: - `frontend/src/api/logs.ts` - `frontend/src/types/logs.ts` **Acceptance Criteria**: - [ ] Can query logs from frontend - [ ] All functions are properly typed --- #### Issue #15: Create Logs Store (Frontend) **Priority**: Medium **Estimated Time**: 2 hours **Tasks**: - Create Zustand store for logs - Implement query state management - Add pagination state - Add filter state **Files to Create**: - `frontend/src/store/logsStore.ts` **Acceptance Criteria**: - [ ] Store manages query state - [ ] Pagination works correctly - [ ] Filters are applied properly --- #### Issue #16: Create Query Builder Component **Priority**: High **Estimated Time**: 4-5 hours **Tasks**: - Create visual query builder UI - Add field selector - Add operator selector - Add value input - Implement query preview - Add raw LogsQL editor toggle **Files to Create**: - `frontend/src/components/logs/QueryBuilder.tsx` **Acceptance Criteria**: - [ ] Can build queries visually - [ ] Can switch to raw LogsQL - [ ] Query preview is accurate --- #### Issue #17: Create Log Table Component **Priority**: High **Estimated Time**: 4 hours **Tasks**: - Create virtualized table (react-window) - Implement expandable rows - Add column customization - Add sorting - Add loading states **Files to Create**: - `frontend/src/components/logs/LogTable.tsx` - `frontend/src/components/logs/LogDetails.tsx` **Acceptance Criteria**: - [ ] Table performs well with 10,000+ rows - [ ] Rows expand to show details - [ ] Columns are customizable --- #### Issue #18: Create Logs Explorer Page **Priority**: High **Estimated Time**: 4 hours **Tasks**: - Create main Logs Explorer page - Integrate QueryBuilder - Integrate LogTable - Add time range selector - Add facets sidebar - Add export button **Files to Create**: - `frontend/src/pages/LogsExplorer.tsx` - `frontend/src/components/logs/TimeRangeSelector.tsx` - `frontend/src/components/logs/FacetsSidebar.tsx` **Acceptance Criteria**: - [ ] Can query and view logs - [ ] Time range selector works - [ ] Facets are clickable filters --- #### Issue #19: Implement Live Tail (WebSocket) **Priority**: Medium **Estimated Time**: 4 hours **Tasks**: - Implement WebSocket endpoint (backend) - Create live tail component (frontend) - Add auto-scroll - Add pause/resume - Add disconnect handling **Files to Create**: - `backend/internal/api/handlers/websocket.go` - `frontend/src/components/logs/LiveTail.tsx` - `frontend/src/hooks/useWebSocket.ts` **Acceptance Criteria**: - [ ] Logs stream in real-time - [ ] Can pause/resume streaming - [ ] Handles disconnections gracefully --- #### Issue #20: Implement Log Export **Priority**: Low **Estimated Time**: 2 hours **Tasks**: - Add CSV export - Add JSON export - Add download handling **Files to Modify**: - `backend/internal/api/handlers/logs.go` - `frontend/src/components/logs/ExportButton.tsx` **Acceptance Criteria**: - [ ] Can export logs as CSV - [ ] Can export logs as JSON - [ ] Large exports stream properly --- ### Milestone 3: Alert Management #### Issue #21: Create Alert Rule Models **Priority**: High **Estimated Time**: 1 hour **Tasks**: - Define alert rule struct - Add JSON/YAML tags - Create validation functions **Files to Create**: - `backend/internal/models/alert.go` **Acceptance Criteria**: - [ ] Alert rule model matches schema - [ ] Validation functions work --- #### Issue #22: Create Alert Repository **Priority**: High **Estimated Time**: 3 hours **Tasks**: - Implement alert rule CRUD - Add database queries - Implement rule search/filter **Files to Create**: - `backend/internal/repository/alert_repo.go` **Acceptance Criteria**: - [ ] Can create/read/update/delete alert rules - [ ] Can filter rules by group/status --- #### Issue #23: Implement vmalert Service **Priority**: High **Estimated Time**: 5 hours **Tasks**: - Create vmalert service - Implement YAML file generation - Implement vmalert reload trigger - Add active alerts fetching - Add validation **Files to Create**: - `backend/internal/services/vmalert_service.go` **Acceptance Criteria**: - [ ] Can generate valid YAML files - [ ] vmalert reloads successfully - [ ] Can fetch active alerts from vmalert --- #### Issue #24: Implement Alert API Endpoints **Priority**: High **Estimated Time**: 3 hours **Tasks**: - Create alert rule CRUD endpoints - Add reload endpoint - Add active alerts endpoint - Add permissions checks **Files to Create**: - `backend/internal/api/handlers/alerts.go` **Acceptance Criteria**: - [ ] All CRUD operations work - [ ] Reload triggers vmalert reload - [ ] Active alerts are fetched --- #### Issue #25: Create Alert API Client (Frontend) **Priority**: High **Estimated Time**: 2 hours **Tasks**: - Create alerts API functions - Add TypeScript types **Files to Create**: - `frontend/src/api/alerts.ts` - `frontend/src/types/alerts.ts` **Acceptance Criteria**: - [ ] Can manage alerts from frontend - [ ] All functions are typed --- #### Issue #26: Create Alert Rule Form Component **Priority**: High **Estimated Time**: 4 hours **Tasks**: - Create alert rule form - Add field validation - Add LogsQL query builder - Add labels/annotations editor **Files to Create**: - `frontend/src/components/alerts/AlertRuleForm.tsx` **Acceptance Criteria**: - [ ] Form validates all fields - [ ] Can create/edit rules - [ ] LogsQL syntax is validated --- #### Issue #27: Create Alert Rule YAML Editor **Priority**: Medium **Estimated Time**: 3 hours **Tasks**: - Integrate Monaco Editor - Add YAML syntax highlighting - Add YAML validation - Add save functionality **Files to Create**: - `frontend/src/components/alerts/AlertRuleEditor.tsx` **Acceptance Criteria**: - [ ] YAML editor has syntax highlighting - [ ] Invalid YAML shows errors - [ ] Can save YAML directly --- #### Issue #28: Create Alerts Manager Page **Priority**: High **Estimated Time**: 4 hours **Tasks**: - Create alerts list view - Add create/edit/delete actions - Add active alerts section - Add alert history **Files to Create**: - `frontend/src/pages/AlertsManager.tsx` - `frontend/src/components/alerts/AlertRuleList.tsx` - `frontend/src/components/alerts/AlertStatus.tsx` **Acceptance Criteria**: - [ ] Can view all alert rules - [ ] Can create/edit/delete rules - [ ] Active alerts are displayed --- ### Milestone 4: Pattern Detection #### Issue #29: Create Pattern Models **Priority**: High **Estimated Time**: 1 hour **Tasks**: - Define pattern struct - Add validation - Create pre-built patterns **Files to Create**: - `backend/internal/models/pattern.go` **Acceptance Criteria**: - [ ] Pattern model is complete - [ ] Pre-built patterns are defined --- #### Issue #30: Create Pattern Repository **Priority**: High **Estimated Time**: 2 hours **Tasks**: - Implement pattern CRUD - Add pattern search **Files to Create**: - `backend/internal/repository/pattern_repo.go` **Acceptance Criteria**: - [ ] Can manage patterns in database --- #### Issue #31: Implement Regex Pattern Detection **Priority**: High **Estimated Time**: 4 hours **Tasks**: - Create pattern detection engine - Implement regex matching - Generate LogsQL queries - Execute detection **Files to Create**: - `backend/internal/services/pattern_service.go` **Acceptance Criteria**: - [ ] Can detect patterns using regex - [ ] Returns matched logs --- #### Issue #32: Implement ML Feature Extraction **Priority**: High **Estimated Time**: 4 hours **Tasks**: - Create feature extractor - Extract temporal features - Extract log content features - Extract numerical features **Files to Create**: - `ml-service/app/utils/feature_extraction.py` **Acceptance Criteria**: - [ ] Extracts relevant features from logs - [ ] Features are properly normalized --- #### Issue #33: Implement ML Anomaly Detector **Priority**: High **Estimated Time**: 5 hours **Tasks**: - Implement Isolation Forest model - Add training functionality - Add prediction functionality - Add model persistence **Files to Create**: - `ml-service/app/models/anomaly_detector.py` **Acceptance Criteria**: - [ ] Model trains successfully - [ ] Model detects anomalies - [ ] Model can be saved/loaded --- #### Issue #34: Implement ML Service Endpoints **Priority**: High **Estimated Time**: 3 hours **Tasks**: - Complete /api/ml/train endpoint - Complete /api/ml/detect endpoint - Add VictoriaLogs integration **Files to Modify**: - `ml-service/app/main.py` - `ml-service/app/api/endpoints.py` **Acceptance Criteria**: - [ ] Training endpoint works - [ ] Detection endpoint works - [ ] Fetches logs from VictoriaLogs --- #### Issue #35: Create ML Service Client (Backend) **Priority**: High **Estimated Time**: 2 hours **Tasks**: - Create ML service HTTP client - Implement train/detect methods **Files to Create**: - `backend/internal/services/ml_service.go` **Acceptance Criteria**: - [ ] Can communicate with ML service - [ ] Handles timeouts properly --- #### Issue #36: Create Pattern Detection Page **Priority**: High **Estimated Time**: 5 hours **Tasks**: - Create pattern library UI - Add regex pattern form - Add ML detection trigger - Display detection results **Files to Create**: - `frontend/src/pages/PatternDetection.tsx` - `frontend/src/components/patterns/PatternLibrary.tsx` - `frontend/src/components/patterns/RegexPatternForm.tsx` - `frontend/src/components/patterns/MLAnomalyViewer.tsx` **Acceptance Criteria**: - [ ] Can browse pattern library - [ ] Can create custom patterns - [ ] Can trigger ML detection - [ ] Results are displayed clearly --- ### Milestone 5: Reports & Dashboards #### Issue #37: Create Report Models **Priority**: High **Estimated Time**: 1 hour **Tasks**: - Define report struct - Define widget configs **Files to Create**: - `backend/internal/models/report.go` **Acceptance Criteria**: - [ ] Report model is complete --- #### Issue #38: Create Report Repository **Priority**: High **Estimated Time**: 2 hours **Tasks**: - Implement report CRUD **Files to Create**: - `backend/internal/repository/report_repo.go` **Acceptance Criteria**: - [ ] Can manage reports --- #### Issue #39: Implement Report Execution Engine **Priority**: High **Estimated Time**: 4 hours **Tasks**: - Create report execution service - Execute queries for widgets - Aggregate data **Files to Create**: - `backend/internal/services/report_service.go` **Acceptance Criteria**: - [ ] Can execute report queries - [ ] Returns formatted data for charts --- #### Issue #40: Integrate ECharts **Priority**: High **Estimated Time**: 2 hours **Tasks**: - Install echarts-for-react - Create chart wrapper components **Files to Create**: - `frontend/src/components/charts/BaseChart.tsx` **Acceptance Criteria**: - [ ] ECharts renders correctly - [ ] Charts are responsive --- #### Issue #41: Create Chart Components **Priority**: High **Estimated Time**: 4 hours **Tasks**: - Create TimeSeriesChart - Create PieChart - Create BarChart - Create HeatmapChart **Files to Create**: - `frontend/src/components/charts/TimeSeriesChart.tsx` - `frontend/src/components/charts/PieChart.tsx` - `frontend/src/components/charts/BarChart.tsx` - `frontend/src/components/charts/HeatmapChart.tsx` **Acceptance Criteria**: - [ ] All chart types render correctly - [ ] Charts are interactive --- #### Issue #42: Create Dashboard Page **Priority**: High **Estimated Time**: 5 hours **Tasks**: - Create dashboard layout - Add summary cards - Add multiple chart widgets - Make layout responsive **Files to Create**: - `frontend/src/pages/Dashboard.tsx` - `frontend/src/components/dashboard/SummaryCard.tsx` - `frontend/src/components/dashboard/DashboardGrid.tsx` **Acceptance Criteria**: - [ ] Dashboard displays key metrics - [ ] Multiple widgets are shown - [ ] Layout is responsive --- #### Issue #43: Create Report Builder UI **Priority**: Medium **Estimated Time**: 5 hours **Tasks**: - Create report builder interface - Add widget configuration - Add query builder for widgets - Add save/load functionality **Files to Create**: - `frontend/src/components/reports/ReportBuilder.tsx` - `frontend/src/components/reports/WidgetConfig.tsx` **Acceptance Criteria**: - [ ] Can create custom reports - [ ] Can configure widgets - [ ] Reports are saved --- ### Milestone 6: Polish & Production #### Issue #44: Implement Rate Limiting **Priority**: High **Estimated Time**: 3 hours **Tasks**: - Add rate limiting middleware - Configure limits per role - Add rate limit headers **Files to Create**: - `backend/internal/api/middleware/ratelimit.go` **Acceptance Criteria**: - [ ] Rate limiting works per role - [ ] Returns 429 when exceeded --- #### Issue #45: Add Input Validation **Priority**: High **Estimated Time**: 3 hours **Tasks**: - Add query validation - Add pattern validation - Add length limits - Add SQL injection prevention **Files to Create**: - `backend/internal/utils/validator.go` **Acceptance Criteria**: - [ ] Dangerous inputs are rejected - [ ] Validation messages are clear --- #### Issue #46: Add Audit Logging **Priority**: High **Estimated Time**: 3 hours **Tasks**: - Create audit logging middleware - Log sensitive operations - Add audit log viewer (admin) **Files to Create**: - `backend/internal/api/middleware/audit.go` - `backend/internal/services/audit_service.go` **Acceptance Criteria**: - [ ] All sensitive operations are logged - [ ] Admins can view audit logs --- #### Issue #47: Add Loading States and Error Handling **Priority**: Medium **Estimated Time**: 4 hours **Tasks**: - Add loading spinners - Add error toast notifications - Improve error messages - Add retry logic **Files to Create**: - `frontend/src/components/common/LoadingSpinner.tsx` - `frontend/src/components/common/Toast.tsx` - `frontend/src/hooks/useToast.ts` **Acceptance Criteria**: - [ ] Loading states show during requests - [ ] Errors display user-friendly messages --- #### Issue #48: Optimize Performance **Priority**: Medium **Estimated Time**: 4 hours **Tasks**: - Add Redis caching (optional) - Optimize database queries - Add frontend code splitting - Optimize bundle size **Acceptance Criteria**: - [ ] Page load time < 2s - [ ] Bundle size < 500KB --- #### Issue #49: Write API Documentation **Priority**: Medium **Estimated Time**: 4 hours **Tasks**: - Document all API endpoints - Add request/response examples - Create Postman collection **Files to Create**: - `docs/api.md` - `docs/postman_collection.json` **Acceptance Criteria**: - [ ] All endpoints are documented - [ ] Examples are provided --- #### Issue #50: Write User Guide **Priority**: Low **Estimated Time**: 3 hours **Tasks**: - Write getting started guide - Document features - Add screenshots **Files to Create**: - `docs/user-guide.md` - `docs/screenshots/` **Acceptance Criteria**: - [ ] User guide is complete - [ ] Screenshots are included --- #### Issue #51: Write Deployment Guide **Priority**: Medium **Estimated Time**: 3 hours **Tasks**: - Document Docker deployment - Document production configuration - Add troubleshooting section **Files to Create**: - `docs/deployment.md` **Acceptance Criteria**: - [ ] Deployment guide is complete - [ ] Production checklist is provided --- #### Issue #52: Add Unit Tests **Priority**: Medium **Estimated Time**: 6 hours **Tasks**: - Write backend unit tests (critical paths) - Write frontend unit tests (critical components) - Add test CI pipeline **Files to Create**: - `backend/*_test.go` - `frontend/src/**/*.test.tsx` - `.github/workflows/test.yml` or `.gitea/workflows/test.yml` **Acceptance Criteria**: - [ ] Critical paths have tests - [ ] Tests pass in CI --- ## Summary **Total Estimated Time**: 160-200 hours (4-5 weeks) **Total Issues**: 52 **Total Milestones**: 6 **Priority Breakdown**: - High Priority: 38 issues - Medium Priority: 11 issues - Low Priority: 3 issues