134 lines
4.8 KiB
Markdown
134 lines
4.8 KiB
Markdown
# Consul Persistence for Connection Monitor
|
|
|
|
This document describes the Consul-based state persistence implementation for the connection monitoring script.
|
|
|
|
## Overview
|
|
|
|
The connection monitor now supports state persistence using Consul's KV store. This allows the script to resume from its previous state if restarted, maintaining continuity of remediation processes and connection state tracking.
|
|
|
|
## Configuration
|
|
|
|
### Consul Server
|
|
- **URL**: `http://consul.service.dc1.consul:8500` (configurable via constructor parameter)
|
|
- **Authentication**: None required (no ACL tokens)
|
|
- **Key Structure**: All state is stored under `qbitcheck/connection_monitor/`
|
|
|
|
### State Data Persisted
|
|
|
|
The following state variables are persisted to Consul:
|
|
|
|
#### Connection State (`state/`)
|
|
- `connection_state`: Current connection state ('stable' or 'unstable')
|
|
- `last_state_change_time`: Timestamp of last state transition
|
|
- `consecutive_failures`: Count of consecutive connection failures
|
|
- `consecutive_stable_checks`: Count of consecutive stable checks
|
|
|
|
#### Remediation State (`remediation/`)
|
|
- `state`: Current remediation phase (None, 'stopping_torrents', 'restarting_nomad', 'waiting_for_stability', 'restarting_torrents')
|
|
- `start_time`: When remediation process started
|
|
- `stabilization_checks`: Count of stabilization checks during remediation
|
|
|
|
#### Stability Tracking (`stability/`)
|
|
- `start_time`: When stability timer started (for 1-hour requirement)
|
|
|
|
#### VPN State (`vpn/`)
|
|
- `vpn_status`: Current VPN status ('running', 'stopped', 'error', 'unknown')
|
|
- `last_vpn_status_change`: Timestamp of last VPN status change
|
|
- `public_ip`: Current public IP address
|
|
- `last_public_ip_change`: Timestamp of last public IP change
|
|
- `public_ip_details`: Geographic and organizational details about the public IP
|
|
|
|
## VPN Monitoring Features
|
|
|
|
The connection monitor now includes comprehensive VPN server monitoring with the following capabilities:
|
|
|
|
### VPN Status Monitoring
|
|
- **Endpoint**: `http://192.168.4.36:8000/v1/vpn/status`
|
|
- **Checks**: VPN service status (returns `{"status":"running"}`)
|
|
- **Frequency**: Every monitoring cycle (default: 30 seconds)
|
|
- **Error Handling**: Retry logic with exponential backoff
|
|
- **State Persistence**: VPN status is tracked in Consul
|
|
|
|
### Public IP Monitoring
|
|
- **Endpoint**: `http://192.168.4.36:8000/v1/publicip/ip`
|
|
- **Checks**: Public IP address and geographic information
|
|
- **Features**:
|
|
- IP change detection with warning logs
|
|
- Geographic location tracking (city, region, country)
|
|
- Organization/ISP information
|
|
- Automatic state persistence
|
|
|
|
### Integration Points
|
|
- VPN status is checked during each monitoring cycle
|
|
- Public IP changes trigger warning-level logs
|
|
- All VPN/IP data is included in debug metrics
|
|
- State is persisted to Consul for continuity
|
|
|
|
## Implementation Details
|
|
|
|
### State Persistence Points
|
|
|
|
State is automatically saved to Consul at these critical points:
|
|
|
|
1. **Connection State Transitions**:
|
|
- When transitioning from stable to unstable (`on_stable_to_unstable`)
|
|
- When transitioning from unstable to stable (`on_unstable_to_stable`)
|
|
|
|
2. **Remediation Process**:
|
|
- When remediation starts (`start_remediation`)
|
|
- After each remediation state transition:
|
|
- Stopping torrents → Restarting Nomad
|
|
- Restarting Nomad → Waiting for stability
|
|
- Waiting for stability → Restarting torrents
|
|
- When remediation completes successfully
|
|
- When remediation fails or times out
|
|
- On unexpected errors during remediation
|
|
|
|
3. **Stability Tracking**:
|
|
- When 1-hour stability requirement is met
|
|
|
|
### Error Handling
|
|
|
|
- If Consul is unavailable, the script continues operation with graceful degradation
|
|
- Consul connection errors are logged but don't interrupt monitoring
|
|
- State loading failures result in default initialization
|
|
|
|
## Usage
|
|
|
|
### Basic Usage
|
|
```python
|
|
monitor = ConnectionMonitor(
|
|
qbittorrent_url='http://sp.service.dc1.consul:8080',
|
|
nomad_url='http://192.168.4.36:4646',
|
|
tracker_name='your_tracker_name',
|
|
consul_url='http://consul.service.dc1.consul:8500' # Optional, defaults to above
|
|
)
|
|
```
|
|
|
|
### Without Consul
|
|
If the `python-consul` package is not installed, state persistence is automatically disabled with a warning message.
|
|
|
|
## Dependencies
|
|
|
|
Add to requirements.txt:
|
|
```
|
|
python-consul>=1.1.0
|
|
```
|
|
|
|
Install with:
|
|
```bash
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
## Benefits
|
|
|
|
1. **State Continuity**: Script can be restarted without losing track of ongoing remediation processes
|
|
2. **Crash Recovery**: Survives process restarts and system reboots
|
|
3. **Monitoring**: External systems can monitor the state via Consul
|
|
4. **Debugging**: Historical state available for troubleshooting
|
|
|
|
## Limitations
|
|
|
|
- Requires Consul server to be available
|
|
- State is eventually consistent (saved after transitions)
|
|
- No built-in state expiration or cleanup (manual Consul management required) |