python v2 - added feartures 1 and 3 - no errors2

2026-01-26 09:02:51 +00:00 · 2025-08-08 14:48:54 -07:00
parent 2da72eec9d
commit b481694ad2
19 changed files with 2387 additions and 0 deletions
--- a/Design.md
+++ b/Design.md
@@ -0,0 +1,727 @@
+# **GarminSync Application Design (Python Version)**
+
+## **Basic Info**
+
+**App Name:** GarminSync  
+**What it does:** A CLI application that downloads `.fit` files for every activity in Garmin Connect.
+
+-----
+
+## **Core Features**
+
+### **CLI Mode (Current)**
+1. List all activities (`garminsync list --all`)
+2. List activities that have not been downloaded (`garminsync list --missing`)
+3. List activities that have been downloaded (`garminsync list --downloaded`)
+4. Download all missing activities (`garminsync download --missing`)
+
+### **Enhanced Features (New)**
+5. **Offline Mode**: List activities without polling Garmin Connect (`garminsync list --missing --offline`)
+6. **Daemon Mode**: Run as background service with scheduled downloads (`garminsync daemon --start`)
+7. **Web UI**: Browser-based interface for daemon monitoring and configuration (`http://localhost:8080`)
+
+-----
+
+## **Tech Stack 🐍**
+
+* **Frontend:** CLI (**Python**)
+* **Backend:** **Python**
+* **Database:** SQLite (`garmin.db`)
+* **Hosting:** Docker container
+* **Key Libraries:**
+    * **`python-garminconnect`**: The library for Garmin Connect API communication.
+    * **`typer`**: A modern and easy-to-use CLI framework (built on `click`).
+    * **`python-dotenv`**: For loading credentials from a `.env` file.
+    * **`sqlalchemy`**: A robust ORM for database interaction and schema management.
+    * **`tqdm`**: For creating user-friendly progress bars.
+    * **`fastapi`**: Modern web framework for the daemon web UI.
+    * **`uvicorn`**: ASGI server for running the FastAPI web interface.
+    * **`apscheduler`**: Advanced Python Scheduler for daemon mode scheduling.
+    * **`pydantic`**: Data validation and settings management for configuration.
+    * **`jinja2`**: Template engine for web UI rendering.
+
+-----
+
+## **Data Structure**
+
+The application uses SQLAlchemy ORM with expanded models for daemon functionality:
+
+**SQLAlchemy Models (`database.py`):**
+
+```python
+class Activity(Base):
+    __tablename__ = 'activities'
+    
+    activity_id = Column(Integer, primary_key=True)
+    start_time = Column(String, nullable=False)
+    filename = Column(String, unique=True, nullable=True)
+    downloaded = Column(Boolean, default=False, nullable=False)
+    created_at = Column(String, nullable=False)  # When record was added
+    last_sync = Column(String, nullable=True)    # Last successful sync
+
+class DaemonConfig(Base):
+    __tablename__ = 'daemon_config'
+    
+    id = Column(Integer, primary_key=True, default=1)
+    enabled = Column(Boolean, default=True, nullable=False)
+    schedule_cron = Column(String, default="0 */6 * * *", nullable=False)  # Every 6 hours
+    last_run = Column(String, nullable=True)
+    next_run = Column(String, nullable=True)
+    status = Column(String, default="stopped", nullable=False)  # stopped, running, error
+
+class SyncLog(Base):
+    __tablename__ = 'sync_logs'
+    
+    id = Column(Integer, primary_key=True, autoincrement=True)
+    timestamp = Column(String, nullable=False)
+    operation = Column(String, nullable=False)  # sync, download, daemon_start, daemon_stop
+    status = Column(String, nullable=False)     # success, error, partial
+    message = Column(String, nullable=True)
+    activities_processed = Column(Integer, default=0, nullable=False)
+    activities_downloaded = Column(Integer, default=0, nullable=False)
+```
+
+-----
+
+## **User Flow**
+
+### **CLI Mode (Existing)**
+1. User sets up credentials in `.env` file with `GARMIN_EMAIL` and `GARMIN_PASSWORD`
+2. User launches the container: `docker run -it --env-file .env -v $(pwd)/data:/app/data garminsync`
+3. User runs commands like `garminsync download --missing`
+4. Application syncs with Garmin Connect, shows progress bars, and downloads activities
+
+### **Offline Mode (New)**
+1. User runs `garminsync list --missing --offline` to view cached data without API calls
+2. Application queries local database only, showing last known state
+3. Useful for checking status without network connectivity or API rate limits
+
+### **Daemon Mode (New)**
+1. User starts daemon: `garminsync daemon --start`
+2. Daemon runs in background, scheduling automatic sync/download operations
+3. User accesses web UI at `http://localhost:8080` for monitoring and configuration
+4. Web UI provides real-time status, logs, and schedule management
+5. Daemon can be stopped with `garminsync daemon --stop` or through web UI
+
+-----
+
+## **File Structure**
+
+```
+/garminsync
+├── garminsync/              # Main application package
+│   ├── __init__.py         # Empty package file
+│   ├── cli.py             # Typer CLI commands and main entrypoint
+│   ├── config.py          # Configuration and environment variable loading
+│   ├── database.py        # SQLAlchemy models and database operations
+│   ├── garmin.py          # Garmin Connect client wrapper with robust download logic
+│   ├── daemon.py          # Daemon mode implementation with APScheduler
+│   ├── web/               # Web UI components
+│   │   ├── __init__.py
+│   │   ├── app.py         # FastAPI application setup
+│   │   ├── routes.py      # API endpoints for web UI
+│   │   ├── static/        # CSS, JavaScript, images
+│   │   │   ├── style.css
+│   │   │   └── app.js
+│   │   └── templates/     # Jinja2 HTML templates
+│   │       ├── base.html
+│   │       ├── dashboard.html
+│   │       └── config.html
+│   └── utils.py           # Shared utilities and helpers
+├── data/                    # Directory for downloaded .fit files and SQLite DB
+├── .env                     # Stores GARMIN_EMAIL/GARMIN_PASSWORD (gitignored)
+├── .gitignore              # Excludes .env file
+├── Dockerfile              # Production-ready container configuration
+├── Design.md               # This design document
+└── requirements.txt        # Pinned Python dependencies (updated)
+```
+
+-----
+
+## **Technical Implementation Details**
+
+### **Architecture**
+- **CLI Framework**: Uses Typer with proper type hints and validation
+- **Module Separation**: Clear separation between CLI commands, database operations, and Garmin API interactions
+- **Error Handling**: Comprehensive exception handling with user-friendly error messages
+- **Session Management**: Proper SQLAlchemy session management with cleanup
+
+### **Authentication & Configuration**
+- Credentials loaded via `python-dotenv` from environment variables
+- Configuration validation ensures required credentials are present
+- Garmin client handles authentication automatically with session persistence
+
+### **Database Operations**
+- SQLite database with SQLAlchemy ORM for type safety
+- Database initialization creates tables automatically
+- Sync functionality reconciles local database with Garmin Connect activities
+- Proper transaction management with rollback on errors
+
+### **File Management**
+- Files named with pattern: `activity_{activity_id}_{timestamp}.fit`
+- Timestamp sanitized for filesystem compatibility (colons and spaces replaced)
+- Downloads saved to configurable data directory
+- Database tracks both download status and file paths
+
+### **API Integration**
+- **Rate Limiting**: 2-second delays between API requests to respect Garmin's servers
+- **Robust Downloads**: Multiple fallback methods for downloading FIT files:
+  1. Default download method
+  2. Explicit 'fit' format parameter
+  3. Alternative parameter names and formats
+  4. Graceful fallback with detailed error reporting
+- **Activity Fetching**: Configurable batch sizes (currently 1000 activities per sync)
+
+### **User Experience**
+- **Progress Indicators**: tqdm progress bars for all long-running operations
+- **Informative Output**: Clear status messages and operation summaries
+- **Input Validation**: Prevents invalid command combinations
+- **Exit Codes**: Proper exit codes for script integration
+
+-----
+
+## **Development Status ✅**
+
+### **✅ Completed Features**
+
+#### **Phase 1: Core Infrastructure**
+- [x] **Dockerfile**: Production-ready Python 3.10 container with proper layer caching
+- [x] **Environment Configuration**: `python-dotenv` integration with validation
+- [x] **CLI Framework**: Complete Typer implementation with type hints and help text
+- [x] **Garmin Integration**: Robust `python-garminconnect` wrapper with authentication
+
+#### **Phase 2: Activity Listing**
+- [x] **Database Schema**: SQLAlchemy models with proper relationships
+- [x] **Database Operations**: Session management, initialization, and sync functionality
+- [x] **List Commands**: All filter options (`--all`, `--missing`, `--downloaded`) implemented
+- [x] **Progress Display**: tqdm integration for user feedback during operations
+
+#### **Phase 3: Download Pipeline**
+- [x] **FIT File Downloads**: Multi-method download approach with fallback strategies
+- [x] **Idempotent Operations**: Prevents re-downloading existing files
+- [x] **Database Updates**: Proper status tracking and file path storage
+- [x] **File Management**: Safe filename generation and directory creation
+
+#### **Phase 4: Polish**
+- [x] **Progress Bars**: Comprehensive tqdm implementation across all operations
+- [x] **Error Handling**: Graceful error handling with informative messages
+- [x] **Container Optimization**: Efficient Docker build with proper dependency management
+
+### **🚧 New Features Implementation Guide**
+
+#### **Feature 1: Offline Mode**
+
+**Implementation Steps:**
+1. **CLI Enhancement** (`cli.py`):
+   ```python
+   @app.command("list")
+   def list_activities(
+       all_activities: bool = False,
+       missing: bool = False,
+       downloaded: bool = False,
+       offline: Annotated[bool, typer.Option("--offline", help="Work offline without syncing")] = False
+   ):
+       if not offline:
+           # Existing sync logic
+           sync_database(client)
+       else:
+           typer.echo("Working in offline mode - using cached data")
+       
+       # Rest of listing logic remains the same
+   ```
+
+2. **Database Enhancements** (`database.py`):
+   - Add `last_sync` column to Activity table
+   - Add utility functions for offline status checking
+   ```python
+   def get_offline_stats():
+       """Return statistics about cached data without API calls"""
+       session = get_session()
+       total = session.query(Activity).count()
+       downloaded = session.query(Activity).filter_by(downloaded=True).count()
+       missing = total - downloaded
+       last_sync = session.query(Activity).order_by(Activity.last_sync.desc()).first()
+       return {
+           'total': total,
+           'downloaded': downloaded, 
+           'missing': missing,
+           'last_sync': last_sync.last_sync if last_sync else 'Never'
+       }
+   ```
+
+#### **Feature 2: Daemon Mode**
+
+**Implementation Steps:**
+1. **New Daemon Module** (`daemon.py`):
+   ```python
+   from apscheduler.schedulers.background import BackgroundScheduler
+   from apscheduler.triggers.cron import CronTrigger
+   import signal
+   import sys
+   import time
+   import threading
+   from datetime import datetime
+   
+   class GarminSyncDaemon:
+       def __init__(self):
+           self.scheduler = BackgroundScheduler()
+           self.running = False
+           self.web_server = None
+           
+       def start(self, web_port=8080):
+           """Start daemon with scheduler and web UI"""
+           # Load configuration from database
+           config = self.load_config()
+           
+           # Setup scheduled job
+           if config.enabled:
+               self.scheduler.add_job(
+                   func=self.sync_and_download,
+                   trigger=CronTrigger.from_crontab(config.schedule_cron),
+                   id='sync_job',
+                   replace_existing=True
+               )
+           
+           # Start scheduler
+           self.scheduler.start()
+           self.running = True
+           
+           # Start web UI in separate thread
+           self.start_web_ui(web_port)
+           
+           # Setup signal handlers for graceful shutdown
+           signal.signal(signal.SIGINT, self.signal_handler)
+           signal.signal(signal.SIGTERM, self.signal_handler)
+           
+           print(f"Daemon started. Web UI available at http://localhost:{web_port}")
+           
+           # Keep daemon running
+           try:
+               while self.running:
+                   time.sleep(1)
+           except KeyboardInterrupt:
+               self.stop()
+               
+       def sync_and_download(self):
+           """Scheduled job function"""
+           try:
+               self.log_operation("sync", "started")
+               
+               # Perform sync and download
+               from .garmin import GarminClient
+               from .database import sync_database
+               
+               client = GarminClient()
+               activities_before = self.count_missing()
+               
+               sync_database(client)
+               
+               # Download missing activities
+               downloaded_count = self.download_missing_activities(client)
+               
+               self.log_operation("sync", "success", 
+                   f"Downloaded {downloaded_count} new activities")
+               
+           except Exception as e:
+               self.log_operation("sync", "error", str(e))
+               
+       def load_config(self):
+           """Load daemon configuration from database"""
+           session = get_session()
+           config = session.query(DaemonConfig).first()
+           if not config:
+               # Create default configuration
+               config = DaemonConfig()
+               session.add(config)
+               session.commit()
+           session.close()
+           return config
+   ```
+
+2. **CLI Integration** (`cli.py`):
+   ```python
+   @app.command("daemon")
+   def daemon_mode(
+       start: Annotated[bool, typer.Option("--start", help="Start daemon")] = False,
+       stop: Annotated[bool, typer.Option("--stop", help="Stop daemon")] = False,
+       status: Annotated[bool, typer.Option("--status", help="Show daemon status")] = False,
+       port: Annotated[int, typer.Option("--port", help="Web UI port")] = 8080
+   ):
+       """Daemon mode operations"""
+       from .daemon import GarminSyncDaemon
+       
+       if start:
+           daemon = GarminSyncDaemon()
+           daemon.start(web_port=port)
+       elif stop:
+           # Implementation for stopping daemon (PID file or signal)
+           pass
+       elif status:
+           # Show current daemon status
+           pass
+   ```
+
+#### **Feature 3: Web UI**
+
+**Implementation Steps:**
+1. **FastAPI Application** (`web/app.py`):
+   ```python
+   from fastapi import FastAPI, Request
+   from fastapi.staticfiles import StaticFiles
+   from fastapi.templating import Jinja2Templates
+   from .routes import router
+   
+   app = FastAPI(title="GarminSync Dashboard")
+   
+   # Mount static files and templates
+   app.mount("/static", StaticFiles(directory="garminsync/web/static"), name="static")
+   templates = Jinja2Templates(directory="garminsync/web/templates")
+   
+   # Include API routes
+   app.include_router(router)
+   
+   @app.get("/")
+   async def dashboard(request: Request):
+       # Get current statistics
+       from ..database import get_offline_stats
+       stats = get_offline_stats()
+       
+       return templates.TemplateResponse("dashboard.html", {
+           "request": request,
+           "stats": stats
+       })
+   ```
+
+2. **API Routes** (`web/routes.py`):
+   ```python
+   from fastapi import APIRouter, HTTPException
+   from pydantic import BaseModel
+   from ..database import get_session, DaemonConfig, SyncLog
+   
+   router = APIRouter(prefix="/api")
+   
+   class ScheduleConfig(BaseModel):
+       enabled: bool
+       cron_schedule: str
+   
+   @router.get("/status")
+   async def get_status():
+       """Get current daemon status"""
+       session = get_session()
+       config = session.query(DaemonConfig).first()
+       
+       # Get recent logs
+       logs = session.query(SyncLog).order_by(SyncLog.timestamp.desc()).limit(10).all()
+       
+       return {
+           "daemon": {
+               "running": config.status == "running" if config else False,
+               "next_run": config.next_run if config else None,
+               "schedule": config.schedule_cron if config else None
+           },
+           "recent_logs": [
+               {
+                   "timestamp": log.timestamp,
+                   "operation": log.operation,
+                   "status": log.status,
+                   "message": log.message
+               } for log in logs
+           ]
+       }
+   
+   @router.post("/schedule")
+   async def update_schedule(config: ScheduleConfig):
+       """Update daemon schedule configuration"""
+       session = get_session()
+       daemon_config = session.query(DaemonConfig).first()
+       
+       if not daemon_config:
+           daemon_config = DaemonConfig()
+           session.add(daemon_config)
+       
+       daemon_config.enabled = config.enabled
+       daemon_config.schedule_cron = config.cron_schedule
+       session.commit()
+       
+       return {"message": "Configuration updated successfully"}
+   
+   @router.post("/sync/trigger")
+   async def trigger_sync():
+       """Manually trigger a sync operation"""
+       # Implementation to trigger immediate sync
+       pass
+   ```
+
+3. **HTML Templates** (`web/templates/dashboard.html`):
+   ```html
+   {% extends "base.html" %}
+   
+   {% block content %}
+   <div class="container">
+       <h1>GarminSync Dashboard</h1>
+       
+       <div class="row">
+           <div class="col-md-4">
+               <div class="card">
+                   <div class="card-header">Statistics</div>
+                   <div class="card-body">
+                       <p>Total Activities: {{ stats.total }}</p>
+                       <p>Downloaded: {{ stats.downloaded }}</p>
+                       <p>Missing: {{ stats.missing }}</p>
+                       <p>Last Sync: {{ stats.last_sync }}</p>
+                   </div>
+               </div>
+           </div>
+           
+           <div class="col-md-4">
+               <div class="card">
+                   <div class="card-header">Daemon Status</div>
+                   <div class="card-body" id="daemon-status">
+                       <!-- Populated by JavaScript -->
+                   </div>
+               </div>
+           </div>
+           
+           <div class="col-md-4">
+               <div class="card">
+                   <div class="card-header">Quick Actions</div>
+                   <div class="card-body">
+                       <button class="btn btn-primary" onclick="triggerSync()">
+                           Sync Now
+                       </button>
+                       <button class="btn btn-secondary" onclick="toggleDaemon()">
+                           Toggle Daemon
+                       </button>
+                   </div>
+               </div>
+           </div>
+       </div>
+       
+       <div class="row mt-4">
+           <div class="col-12">
+               <div class="card">
+                   <div class="card-header">Recent Activity</div>
+                   <div class="card-body" id="recent-logs">
+                       <!-- Populated by JavaScript -->
+                   </div>
+               </div>
+           </div>
+       </div>
+       
+       <div class="row mt-4">
+           <div class="col-12">
+               <div class="card">
+                   <div class="card-header">Schedule Configuration</div>
+                   <div class="card-body">
+                       <form id="schedule-form">
+                           <div class="form-group">
+                               <label for="schedule-enabled">Enable Scheduled Sync</label>
+                               <input type="checkbox" id="schedule-enabled">
+                           </div>
+                           <div class="form-group">
+                               <label for="cron-schedule">Cron Schedule</label>
+                               <input type="text" class="form-control" id="cron-schedule" 
+                                      placeholder="0 */6 * * *" title="Every 6 hours">
+                           </div>
+                           <button type="submit" class="btn btn-primary">
+                               Update Schedule
+                           </button>
+                       </form>
+                   </div>
+               </div>
+           </div>
+       </div>
+   </div>
+   {% endblock %}
+   ```
+
+4. **JavaScript for Interactivity** (`web/static/app.js`):
+   ```javascript
+   // Auto-refresh dashboard data
+   setInterval(updateStatus, 30000); // Every 30 seconds
+   
+   async function updateStatus() {
+       try {
+           const response = await fetch('/api/status');
+           const data = await response.json();
+           
+           // Update daemon status
+           document.getElementById('daemon-status').innerHTML = `
+               <p>Status: <span class="badge ${data.daemon.running ? 'badge-success' : 'badge-danger'}">
+                   ${data.daemon.running ? 'Running' : 'Stopped'}
+               </span></p>
+               <p>Next Run: ${data.daemon.next_run || 'Not scheduled'}</p>
+               <p>Schedule: ${data.daemon.schedule || 'Not configured'}</p>
+           `;
+           
+           // Update recent logs
+           const logsHtml = data.recent_logs.map(log => `
+               <div class="log-entry">
+                   <small class="text-muted">${log.timestamp}</small>
+                   <span class="badge badge-${log.status === 'success' ? 'success' : 'danger'}">
+                       ${log.status}
+                   </span>
+                   ${log.operation}: ${log.message || ''}
+               </div>
+           `).join('');
+           
+           document.getElementById('recent-logs').innerHTML = logsHtml;
+           
+       } catch (error) {
+           console.error('Failed to update status:', error);
+       }
+   }
+   
+   async function triggerSync() {
+       try {
+           await fetch('/api/sync/trigger', { method: 'POST' });
+           alert('Sync triggered successfully');
+           updateStatus();
+       } catch (error) {
+           alert('Failed to trigger sync');
+       }
+   }
+   
+   // Initialize on page load
+   document.addEventListener('DOMContentLoaded', updateStatus);
+   ```
+
+### **Updated Requirements** (`requirements.txt`):
+```
+typer==0.9.0
+click==8.1.7
+python-dotenv==1.0.0
+garminconnect==0.2.28
+sqlalchemy==2.0.23
+tqdm==4.66.1
+fastapi==0.104.1
+uvicorn[standard]==0.24.0
+apscheduler==3.10.4
+pydantic==2.5.0
+jinja2==3.1.2
+python-multipart==0.0.6
+aiofiles==23.2.1
+```
+
+### **Docker Updates**:
+```dockerfile
+# Expose web UI port
+EXPOSE 8080
+
+# Update entrypoint to support daemon mode
+ENTRYPOINT ["python", "-m", "garminsync.cli"]
+CMD ["--help"]
+```
+
+### **Usage Examples**:
+
+**Offline Mode:**
+```bash
+# List missing activities without network calls
+docker run --env-file .env -v $(pwd)/data:/app/data garminsync list --missing --offline
+```
+
+**Daemon Mode:**
+```bash
+# Start daemon with web UI on port 8080
+docker run -d --env-file .env -v $(pwd)/data:/app/data -p 8080:8080 garminsync daemon --start
+
+# Access web UI at http://localhost:8080
+```
+
+-----
+
+## **Docker Usage**
+
+### **Build the Container**
+```bash
+docker build -t garminsync .
+```
+
+### **Run with Environment File**
+```bash
+docker run -it --env-file .env -v $(pwd)/data:/app/data garminsync --help
+```
+
+### **Example Commands**
+```bash
+# List all activities
+docker run -it --env-file .env -v $(pwd)/data:/app/data garminsync list --all
+
+# Download missing activities
+docker run -it --env-file .env -v $(pwd)/data:/app/data garminsync download --missing
+```
+
+-----
+
+## **Environment Setup**
+
+Create a `.env` file in the project root:
+```
+GARMIN_EMAIL=your_email@example.com
+GARMIN_PASSWORD=your_password
+```
+
+-----
+
+## **Key Implementation Highlights**
+
+### **Robust Download Logic**
+The `garmin.py` module implements a sophisticated download strategy that tries multiple methods to handle variations in the Garmin Connect API:
+
+```python
+methods_to_try = [
+    lambda: self.client.download_activity(activity_id),
+    lambda: self.client.download_activity(activity_id, fmt='fit'),
+    lambda: self.client.download_activity(activity_id, format='fit'),
+    # ... additional fallback methods
+]
+```
+
+### **Database Synchronization**
+The sync process efficiently updates the local database with new activities from Garmin Connect:
+
+```python
+def sync_database(garmin_client):
+    """Sync local database with Garmin Connect activities"""
+    activities = garmin_client.get_activities(0, 1000)
+    for activity in activities:
+        # Only add new activities, preserve existing download status
+        existing = session.query(Activity).filter_by(activity_id=activity_id).first()
+        if not existing:
+            new_activity = Activity(...)
+            session.add(new_activity)
+```
+
+### **CLI Integration**
+Clean separation between CLI interface and business logic with proper type annotations:
+
+```python
+def list_activities(
+    all_activities: Annotated[bool, typer.Option("--all", help="List all activities")] = False,
+    missing: Annotated[bool, typer.Option("--missing", help="List missing activities")] = False,
+    downloaded: Annotated[bool, typer.Option("--downloaded", help="List downloaded activities")] = False
+):
+```
+
+-----
+
+## **Documentation 📚**
+
+Here are links to the official documentation for the key libraries used:
+
+* **Garmin API:** [python-garminconnect](https://github.com/cyberjunky/python-garminconnect)
+* **CLI Framework:** [Typer](https://typer.tiangolo.com/)
+* **Environment Variables:** [python-dotenv](https://github.com/theskumar/python-dotenv)
+* **Database ORM:** [SQLAlchemy](https://docs.sqlalchemy.org/en/20/)
+* **Progress Bars:** [tqdm](https://github.com/tqdm/tqdm)
+
+-----
+
+## **Performance Considerations**
+
+- **Rate Limiting**: 2-second delays between API requests prevent server overload
+- **Batch Processing**: Fetches up to 1000 activities per sync operation
+- **Efficient Queries**: Database queries optimized for filtering operations
+- **Memory Management**: Proper session cleanup and resource management
+- **Docker Optimization**: Layer caching and minimal base image for faster builds