victorialogs-manager/IMPLEMENTATION_PLAN.md

VictoriaLogs Manager - Implementation Plan

Project Overview

A comprehensive web application for managing VictoriaLogs with alerting, advanced reporting, and malicious pattern detection.

Milestones & Phases

Milestone 1: Foundation & Authentication (Week 1)

Goal: Complete project setup and implement authentication system

Duration: 5-7 days

Deliverables:

• Fully configured development environment
• Working JWT authentication
• User management system
• Role-based access control

---

Milestone 2: VictoriaLogs Integration (Week 2)

Goal: Implement log querying and exploration features

Duration: 5-7 days

Deliverables:

• VictoriaLogs HTTP client
• Log query API endpoints
• Logs Explorer UI
• Live tail functionality
• Export features

---

Milestone 3: Alert Management (Week 3)

Goal: Complete vmalert rule management system

Duration: 5-7 days

Deliverables:

• Alert rule CRUD operations
• YAML file generation
• vmalert integration
• Alert Manager UI
• Active alerts viewer

---

Milestone 4: Pattern Detection (Week 4)

Goal: Implement regex and ML-based pattern detection

Duration: 7-10 days

Deliverables:

• Regex pattern engine
• Pre-built pattern library
• ML service implementation
• Pattern Detection UI
• ML training and detection endpoints

---

Milestone 5: Reports & Dashboards (Week 5)

Goal: Build visualization and reporting system

Duration: 5-7 days

Deliverables:

• Report configuration system
• Dashboard page with widgets
• ECharts integration
• Report builder UI
• Multiple chart types

---

Milestone 6: Polish & Production (Week 6)

Goal: Production readiness and deployment

Duration: 5-7 days

Deliverables:

• Security hardening
• Performance optimization
• Documentation
• Testing
• Deployment guides

---

Detailed Task Breakdown

Milestone 1: Foundation & Authentication

Issue #1: Setup React Frontend

Priority: High Estimated Time: 2-3 hours

Tasks:

• Initialize Vite + React + TypeScript project
• Install dependencies (React Router, Zustand, Axios, Tailwind CSS)
• Configure Tailwind CSS
• Create basic folder structure
• Setup ESLint and Prettier

Files to Create:

• frontend/package.json
• frontend/vite.config.ts
• frontend/tailwind.config.js
• frontend/src/main.tsx
• frontend/src/App.tsx

Acceptance Criteria:

• Frontend dev server runs successfully
• Tailwind CSS is working
• Hot reload is functional

---

Issue #2: Implement JWT Utilities

Priority: High Estimated Time: 2 hours

Tasks:

• Create JWT token generation function
• Create JWT token validation function
• Implement claims structure
• Add token expiration handling

Files to Create:

• backend/internal/utils/jwt.go

Acceptance Criteria:

• Can generate JWT tokens with user claims
• Can validate and parse JWT tokens
• Token expiration is enforced

---

Issue #3: Implement Authentication Handlers

Priority: High Estimated Time: 3-4 hours

Tasks:

• Create login handler (username/password)
• Create refresh token handler
• Create "get current user" handler
• Implement password hashing with bcrypt
• Add input validation

Files to Create:

• backend/internal/api/handlers/auth.go
• backend/internal/services/auth_service.go

Acceptance Criteria:

• POST /api/v1/auth/login returns JWT token
• POST /api/v1/auth/refresh refreshes token
• GET /api/v1/auth/me returns current user
• Passwords are securely hashed

---

Issue #4: Implement Authentication Middleware

Priority: High Estimated Time: 2 hours

Tasks:

• Create JWT validation middleware
• Create permission check middleware
• Create role check middleware
• Add error handling

Files to Create:

• backend/internal/api/middleware/auth.go

Acceptance Criteria:

• Middleware validates JWT tokens
• Invalid tokens return 401
• Permission checks work correctly
• Role checks work correctly

---

Issue #5: Create User Repository

Priority: High Estimated Time: 2-3 hours

Tasks:

• Implement user CRUD operations
• Create database queries
• Add error handling
• Implement user lookup functions

Files to Create:

• backend/internal/repository/user_repo.go

Acceptance Criteria:

• Can create new users
• Can retrieve users by ID/username/email
• Can update user information
• Can delete users
• Can list all users

---

Issue #6: Implement User Management API

Priority: High Estimated Time: 3 hours

Tasks:

• Create user CRUD endpoints
• Add role-based access control
• Implement user service layer
• Add input validation

Files to Create:

• backend/internal/api/handlers/users.go
• backend/internal/services/user_service.go

Acceptance Criteria:

• GET /api/v1/users returns user list (admin only)
• POST /api/v1/users creates user (admin only)
• PUT /api/v1/users/:id updates user (admin only)
• DELETE /api/v1/users/:id deletes user (admin only)

---

Issue #7: Create Auth Store (Frontend)

Priority: High Estimated Time: 2 hours

Tasks:

• Setup Zustand store for authentication
• Implement login/logout functions
• Add token persistence (localStorage)
• Create auth hooks

Files to Create:

• frontend/src/store/authStore.ts
• frontend/src/hooks/useAuth.ts

Acceptance Criteria:

• Auth state persists across page reloads
• Login updates auth state
• Logout clears auth state

---

Issue #8: Create API Client (Frontend)

Priority: High Estimated Time: 2 hours

Tasks:

• Setup Axios instance
• Add request interceptor (JWT token)
• Add response interceptor (401 handling)
• Create API client functions

Files to Create:

• frontend/src/api/client.ts
• frontend/src/api/auth.ts

Acceptance Criteria:

• All requests include JWT token
• 401 responses trigger logout
• API client is properly typed

---

Issue #9: Create Login Page

Priority: High Estimated Time: 3 hours

Tasks:

• Create login form UI
• Add form validation
• Implement login submission
• Add error handling
• Add loading states

Files to Create:

• frontend/src/pages/Login.tsx

Acceptance Criteria:

• Login form validates inputs
• Successful login redirects to dashboard
• Error messages display correctly
• Loading state shows during request

---

Issue #10: Create Protected Route Component

Priority: High Estimated Time: 1 hour

Tasks:

• Create ProtectedRoute component
• Add authentication check
• Add role-based access
• Implement redirects

Files to Create:

• frontend/src/components/common/ProtectedRoute.tsx

Acceptance Criteria:

• Unauthenticated users redirect to login
• Unauthorized users see error message
• Authorized users see content

---

Milestone 2: VictoriaLogs Integration

Issue #11: Create VictoriaLogs HTTP Client

Priority: High Estimated Time: 4 hours

Tasks:

• Create VictoriaLogs HTTP client
• Implement query method
• Implement tail method
• Implement facets method
• Implement stats query method
• Add error handling

Files to Create:

• backend/pkg/vlogs/client.go
• backend/pkg/vlogs/models.go

Acceptance Criteria:

• Can query VictoriaLogs
• Can tail logs in real-time
• Can fetch facets
• Can execute stats queries

---

Issue #12: Create Log Query Service

Priority: High Estimated Time: 3 hours

Tasks:

• Implement log query proxy
• Add authentication checks
• Implement rate limiting
• Add query validation

Files to Create:

• backend/internal/services/vlogs_service.go

Acceptance Criteria:

• Service proxies requests to VictoriaLogs
• User permissions are enforced
• Rate limiting works per role

---

Issue #13: Implement Log Query API Endpoints

Priority: High Estimated Time: 3 hours

Tasks:

• Create POST /logs/query endpoint
• Create POST /logs/tail endpoint
• Create GET /logs/facets endpoint
• Create POST /logs/stats endpoint
• Create POST /logs/export endpoint

Files to Create:

• backend/internal/api/handlers/logs.go

Acceptance Criteria:

• All endpoints return correct data
• Authentication is required
• Error handling is robust

---

Issue #14: Create Logs API Client (Frontend)

Priority: High Estimated Time: 2 hours

Tasks:

• Create logs API functions
• Add TypeScript types
• Implement query builder helper

Files to Create:

• frontend/src/api/logs.ts
• frontend/src/types/logs.ts

Acceptance Criteria:

• Can query logs from frontend
• All functions are properly typed

---

Issue #15: Create Logs Store (Frontend)

Priority: Medium Estimated Time: 2 hours

Tasks:

• Create Zustand store for logs
• Implement query state management
• Add pagination state
• Add filter state

Files to Create:

• frontend/src/store/logsStore.ts

Acceptance Criteria:

• Store manages query state
• Pagination works correctly
• Filters are applied properly

---

Issue #16: Create Query Builder Component

Priority: High Estimated Time: 4-5 hours

Tasks:

• Create visual query builder UI
• Add field selector
• Add operator selector
• Add value input
• Implement query preview
• Add raw LogsQL editor toggle

Files to Create:

• frontend/src/components/logs/QueryBuilder.tsx

Acceptance Criteria:

• Can build queries visually
• Can switch to raw LogsQL
• Query preview is accurate

---

Issue #17: Create Log Table Component

Priority: High Estimated Time: 4 hours

Tasks:

• Create virtualized table (react-window)
• Implement expandable rows
• Add column customization
• Add sorting
• Add loading states

Files to Create:

• frontend/src/components/logs/LogTable.tsx
• frontend/src/components/logs/LogDetails.tsx

Acceptance Criteria:

• Table performs well with 10,000+ rows
• Rows expand to show details
• Columns are customizable

---

Issue #18: Create Logs Explorer Page

Priority: High Estimated Time: 4 hours

Tasks:

• Create main Logs Explorer page
• Integrate QueryBuilder
• Integrate LogTable
• Add time range selector
• Add facets sidebar
• Add export button

Files to Create:

• frontend/src/pages/LogsExplorer.tsx
• frontend/src/components/logs/TimeRangeSelector.tsx
• frontend/src/components/logs/FacetsSidebar.tsx

Acceptance Criteria:

• Can query and view logs
• Time range selector works
• Facets are clickable filters

---

Issue #19: Implement Live Tail (WebSocket)

Priority: Medium Estimated Time: 4 hours

Tasks:

• Implement WebSocket endpoint (backend)
• Create live tail component (frontend)
• Add auto-scroll
• Add pause/resume
• Add disconnect handling

Files to Create:

• backend/internal/api/handlers/websocket.go
• frontend/src/components/logs/LiveTail.tsx
• frontend/src/hooks/useWebSocket.ts

Acceptance Criteria:

• Logs stream in real-time
• Can pause/resume streaming
• Handles disconnections gracefully

---

Issue #20: Implement Log Export

Priority: Low Estimated Time: 2 hours

Tasks:

• Add CSV export
• Add JSON export
• Add download handling

Files to Modify:

• backend/internal/api/handlers/logs.go
• frontend/src/components/logs/ExportButton.tsx

Acceptance Criteria:

• Can export logs as CSV
• Can export logs as JSON
• Large exports stream properly

---

Milestone 3: Alert Management

Issue #21: Create Alert Rule Models

Priority: High Estimated Time: 1 hour

Tasks:

• Define alert rule struct
• Add JSON/YAML tags
• Create validation functions

Files to Create:

• backend/internal/models/alert.go

Acceptance Criteria:

• Alert rule model matches schema
• Validation functions work

---

Issue #22: Create Alert Repository

Priority: High Estimated Time: 3 hours

Tasks:

• Implement alert rule CRUD
• Add database queries
• Implement rule search/filter

Files to Create:

• backend/internal/repository/alert_repo.go

Acceptance Criteria:

• Can create/read/update/delete alert rules
• Can filter rules by group/status

---

Issue #23: Implement vmalert Service

Priority: High Estimated Time: 5 hours

Tasks:

• Create vmalert service
• Implement YAML file generation
• Implement vmalert reload trigger
• Add active alerts fetching
• Add validation

Files to Create:

• backend/internal/services/vmalert_service.go

Acceptance Criteria:

• Can generate valid YAML files
• vmalert reloads successfully
• Can fetch active alerts from vmalert

---

Issue #24: Implement Alert API Endpoints

Priority: High Estimated Time: 3 hours

Tasks:

• Create alert rule CRUD endpoints
• Add reload endpoint
• Add active alerts endpoint
• Add permissions checks

Files to Create:

• backend/internal/api/handlers/alerts.go

Acceptance Criteria:

• All CRUD operations work
• Reload triggers vmalert reload
• Active alerts are fetched

---

Issue #25: Create Alert API Client (Frontend)

Priority: High Estimated Time: 2 hours

Tasks:

• Create alerts API functions
• Add TypeScript types

Files to Create:

• frontend/src/api/alerts.ts
• frontend/src/types/alerts.ts

Acceptance Criteria:

• Can manage alerts from frontend
• All functions are typed

---

Issue #26: Create Alert Rule Form Component

Priority: High Estimated Time: 4 hours

Tasks:

• Create alert rule form
• Add field validation
• Add LogsQL query builder
• Add labels/annotations editor

Files to Create:

• frontend/src/components/alerts/AlertRuleForm.tsx

Acceptance Criteria:

• Form validates all fields
• Can create/edit rules
• LogsQL syntax is validated

---

Issue #27: Create Alert Rule YAML Editor

Priority: Medium Estimated Time: 3 hours

Tasks:

• Integrate Monaco Editor
• Add YAML syntax highlighting
• Add YAML validation
• Add save functionality

Files to Create:

• frontend/src/components/alerts/AlertRuleEditor.tsx

Acceptance Criteria:

• YAML editor has syntax highlighting
• Invalid YAML shows errors
• Can save YAML directly

---

Issue #28: Create Alerts Manager Page

Priority: High Estimated Time: 4 hours

Tasks:

• Create alerts list view
• Add create/edit/delete actions
• Add active alerts section
• Add alert history

Files to Create:

• frontend/src/pages/AlertsManager.tsx
• frontend/src/components/alerts/AlertRuleList.tsx
• frontend/src/components/alerts/AlertStatus.tsx

Acceptance Criteria:

• Can view all alert rules
• Can create/edit/delete rules
• Active alerts are displayed

---

Milestone 4: Pattern Detection

Issue #29: Create Pattern Models

Priority: High Estimated Time: 1 hour

Tasks:

• Define pattern struct
• Add validation
• Create pre-built patterns

Files to Create:

• backend/internal/models/pattern.go

Acceptance Criteria:

• Pattern model is complete
• Pre-built patterns are defined

---

Issue #30: Create Pattern Repository

Priority: High Estimated Time: 2 hours

Tasks:

• Implement pattern CRUD
• Add pattern search

Files to Create:

• backend/internal/repository/pattern_repo.go

Acceptance Criteria:

• Can manage patterns in database

---

Issue #31: Implement Regex Pattern Detection

Priority: High Estimated Time: 4 hours

Tasks:

• Create pattern detection engine
• Implement regex matching
• Generate LogsQL queries
• Execute detection

Files to Create:

• backend/internal/services/pattern_service.go

Acceptance Criteria:

• Can detect patterns using regex
• Returns matched logs

---

Issue #32: Implement ML Feature Extraction

Priority: High Estimated Time: 4 hours

Tasks:

• Create feature extractor
• Extract temporal features
• Extract log content features
• Extract numerical features

Files to Create:

• ml-service/app/utils/feature_extraction.py

Acceptance Criteria:

• Extracts relevant features from logs
• Features are properly normalized

---

Issue #33: Implement ML Anomaly Detector

Priority: High Estimated Time: 5 hours

Tasks:

• Implement Isolation Forest model
• Add training functionality
• Add prediction functionality
• Add model persistence

Files to Create:

• ml-service/app/models/anomaly_detector.py

Acceptance Criteria:

• Model trains successfully
• Model detects anomalies
• Model can be saved/loaded

---

Issue #34: Implement ML Service Endpoints

Priority: High Estimated Time: 3 hours

Tasks:

• Complete /api/ml/train endpoint
• Complete /api/ml/detect endpoint
• Add VictoriaLogs integration

Files to Modify:

• ml-service/app/main.py
• ml-service/app/api/endpoints.py

Acceptance Criteria:

• Training endpoint works
• Detection endpoint works
• Fetches logs from VictoriaLogs

---

Issue #35: Create ML Service Client (Backend)

Priority: High Estimated Time: 2 hours

Tasks:

• Create ML service HTTP client
• Implement train/detect methods

Files to Create:

• backend/internal/services/ml_service.go

Acceptance Criteria:

• Can communicate with ML service
• Handles timeouts properly

---

Issue #36: Create Pattern Detection Page

Priority: High Estimated Time: 5 hours

Tasks:

• Create pattern library UI
• Add regex pattern form
• Add ML detection trigger
• Display detection results

Files to Create:

• frontend/src/pages/PatternDetection.tsx
• frontend/src/components/patterns/PatternLibrary.tsx
• frontend/src/components/patterns/RegexPatternForm.tsx
• frontend/src/components/patterns/MLAnomalyViewer.tsx

Acceptance Criteria:

• Can browse pattern library
• Can create custom patterns
• Can trigger ML detection
• Results are displayed clearly

---

Milestone 5: Reports & Dashboards

Issue #37: Create Report Models

Priority: High Estimated Time: 1 hour

Tasks:

• Define report struct
• Define widget configs

Files to Create:

• backend/internal/models/report.go

Acceptance Criteria:

• Report model is complete

---

Issue #38: Create Report Repository

Priority: High Estimated Time: 2 hours

Tasks:

• Implement report CRUD

Files to Create:

• backend/internal/repository/report_repo.go

Acceptance Criteria:

• Can manage reports

---

Issue #39: Implement Report Execution Engine

Priority: High Estimated Time: 4 hours

Tasks:

• Create report execution service
• Execute queries for widgets
• Aggregate data

Files to Create:

• backend/internal/services/report_service.go

Acceptance Criteria:

• Can execute report queries
• Returns formatted data for charts

---

Issue #40: Integrate ECharts

Priority: High Estimated Time: 2 hours

Tasks:

• Install echarts-for-react
• Create chart wrapper components

Files to Create:

• frontend/src/components/charts/BaseChart.tsx

Acceptance Criteria:

• ECharts renders correctly
• Charts are responsive

---

Issue #41: Create Chart Components

Priority: High Estimated Time: 4 hours

Tasks:

• Create TimeSeriesChart
• Create PieChart
• Create BarChart
• Create HeatmapChart

Files to Create:

• frontend/src/components/charts/TimeSeriesChart.tsx
• frontend/src/components/charts/PieChart.tsx
• frontend/src/components/charts/BarChart.tsx
• frontend/src/components/charts/HeatmapChart.tsx

Acceptance Criteria:

• All chart types render correctly
• Charts are interactive

---

Issue #42: Create Dashboard Page

Priority: High Estimated Time: 5 hours

Tasks:

• Create dashboard layout
• Add summary cards
• Add multiple chart widgets
• Make layout responsive

Files to Create:

• frontend/src/pages/Dashboard.tsx
• frontend/src/components/dashboard/SummaryCard.tsx
• frontend/src/components/dashboard/DashboardGrid.tsx

Acceptance Criteria:

• Dashboard displays key metrics
• Multiple widgets are shown
• Layout is responsive

---

Issue #43: Create Report Builder UI

Priority: Medium Estimated Time: 5 hours

Tasks:

• Create report builder interface
• Add widget configuration
• Add query builder for widgets
• Add save/load functionality

Files to Create:

• frontend/src/components/reports/ReportBuilder.tsx
• frontend/src/components/reports/WidgetConfig.tsx

Acceptance Criteria:

• Can create custom reports
• Can configure widgets
• Reports are saved

---

Milestone 6: Polish & Production

Issue #44: Implement Rate Limiting

Priority: High Estimated Time: 3 hours

Tasks:

• Add rate limiting middleware
• Configure limits per role
• Add rate limit headers

Files to Create:

• backend/internal/api/middleware/ratelimit.go

Acceptance Criteria:

• Rate limiting works per role
• Returns 429 when exceeded

---

Issue #45: Add Input Validation

Priority: High Estimated Time: 3 hours

Tasks:

• Add query validation
• Add pattern validation
• Add length limits
• Add SQL injection prevention

Files to Create:

• backend/internal/utils/validator.go

Acceptance Criteria:

• Dangerous inputs are rejected
• Validation messages are clear

---

Issue #46: Add Audit Logging

Priority: High Estimated Time: 3 hours

Tasks:

• Create audit logging middleware
• Log sensitive operations
• Add audit log viewer (admin)

Files to Create:

• backend/internal/api/middleware/audit.go
• backend/internal/services/audit_service.go

Acceptance Criteria:

• All sensitive operations are logged
• Admins can view audit logs

---

Issue #47: Add Loading States and Error Handling

Priority: Medium Estimated Time: 4 hours

Tasks:

• Add loading spinners
• Add error toast notifications
• Improve error messages
• Add retry logic

Files to Create:

• frontend/src/components/common/LoadingSpinner.tsx
• frontend/src/components/common/Toast.tsx
• frontend/src/hooks/useToast.ts

Acceptance Criteria:

• Loading states show during requests
• Errors display user-friendly messages

---

Issue #48: Optimize Performance

Priority: Medium Estimated Time: 4 hours

Tasks:

• Add Redis caching (optional)
• Optimize database queries
• Add frontend code splitting
• Optimize bundle size

Acceptance Criteria:

• Page load time < 2s
• Bundle size < 500KB

---

Issue #49: Write API Documentation

Priority: Medium Estimated Time: 4 hours

Tasks:

• Document all API endpoints
• Add request/response examples
• Create Postman collection

Files to Create:

• docs/api.md
• docs/postman_collection.json

Acceptance Criteria:

• All endpoints are documented
• Examples are provided

---

Issue #50: Write User Guide

Priority: Low Estimated Time: 3 hours

Tasks:

• Write getting started guide
• Document features
• Add screenshots

Files to Create:

• docs/user-guide.md
• docs/screenshots/

Acceptance Criteria:

• User guide is complete
• Screenshots are included

---

Issue #51: Write Deployment Guide

Priority: Medium Estimated Time: 3 hours

Tasks:

• Document Docker deployment
• Document production configuration
• Add troubleshooting section

Files to Create:

• docs/deployment.md

Acceptance Criteria:

• Deployment guide is complete
• Production checklist is provided

---

Issue #52: Add Unit Tests

Priority: Medium Estimated Time: 6 hours

Tasks:

• Write backend unit tests (critical paths)
• Write frontend unit tests (critical components)
• Add test CI pipeline

Files to Create:

• backend/*_test.go
• frontend/src/**/*.test.tsx
• .github/workflows/test.yml or .gitea/workflows/test.yml

Acceptance Criteria:

• Critical paths have tests
• Tests pass in CI

---

Summary

Total Estimated Time: 160-200 hours (4-5 weeks)

Total Issues: 52

Total Milestones: 6

Priority Breakdown:

• High Priority: 38 issues
• Medium Priority: 11 issues
• Low Priority: 3 issues