GarminSync/Design.md

GarminSync Application Design (Python Version)

Basic Info

App Name: GarminSync
What it does: A CLI application that downloads .fit files for every activity in Garmin Connect.

---

Core Features

CLI Mode (Current)

1. List all activities (garminsync list --all)
2. List activities that have not been downloaded (garminsync list --missing)
3. List activities that have been downloaded (garminsync list --downloaded)
4. Download all missing activities (garminsync download --missing)

Enhanced Features (New)

5. Offline Mode: List activities without polling Garmin Connect (garminsync list --missing --offline)
6. Daemon Mode: Run as background service with scheduled downloads (garminsync daemon --start)
7. Web UI: Browser-based interface for daemon monitoring and configuration (http://localhost:8080)

---

Tech Stack 🐍

• Frontend: CLI (Python)
• Backend: Python
• Database: SQLite (garmin.db)
• Hosting: Docker container
• Key Libraries:
  • python-garminconnect: The library for Garmin Connect API communication.
  • typer: A modern and easy-to-use CLI framework (built on click).
  • python-dotenv: For loading credentials from a .env file.
  • sqlalchemy: A robust ORM for database interaction and schema management.
  • tqdm: For creating user-friendly progress bars.
  • fastapi: Modern web framework for the daemon web UI.
  • uvicorn: ASGI server for running the FastAPI web interface.
  • apscheduler: Advanced Python Scheduler for daemon mode scheduling.
  • pydantic: Data validation and settings management for configuration.
  • jinja2: Template engine for web UI rendering.

---

Data Structure

The application uses SQLAlchemy ORM with expanded models for daemon functionality:

SQLAlchemy Models (database.py):

class Activity(Base):
    __tablename__ = 'activities'
    
    activity_id = Column(Integer, primary_key=True)
    start_time = Column(String, nullable=False)
    filename = Column(String, unique=True, nullable=True)
    downloaded = Column(Boolean, default=False, nullable=False)
    created_at = Column(String, nullable=False)  # When record was added
    last_sync = Column(String, nullable=True)    # Last successful sync

class DaemonConfig(Base):
    __tablename__ = 'daemon_config'
    
    id = Column(Integer, primary_key=True, default=1)
    enabled = Column(Boolean, default=True, nullable=False)
    schedule_cron = Column(String, default="0 */6 * * *", nullable=False)  # Every 6 hours
    last_run = Column(String, nullable=True)
    next_run = Column(String, nullable=True)
    status = Column(String, default="stopped", nullable=False)  # stopped, running, error

class SyncLog(Base):
    __tablename__ = 'sync_logs'
    
    id = Column(Integer, primary_key=True, autoincrement=True)
    timestamp = Column(String, nullable=False)
    operation = Column(String, nullable=False)  # sync, download, daemon_start, daemon_stop
    status = Column(String, nullable=False)     # success, error, partial
    message = Column(String, nullable=True)
    activities_processed = Column(Integer, default=0, nullable=False)
    activities_downloaded = Column(Integer, default=0, nullable=False)

---

User Flow

CLI Mode (Existing)

1. User sets up credentials in .env file with GARMIN_EMAIL and GARMIN_PASSWORD
2. User launches the container: docker run -it --env-file .env -v $(pwd)/data:/app/data garminsync
3. User runs commands like garminsync download --missing
4. Application syncs with Garmin Connect, shows progress bars, and downloads activities

Offline Mode (New)

1. User runs garminsync list --missing --offline to view cached data without API calls
2. Application queries local database only, showing last known state
3. Useful for checking status without network connectivity or API rate limits

Daemon Mode (New)

1. User starts daemon: garminsync daemon --start
2. Daemon runs in background, scheduling automatic sync/download operations
3. User accesses web UI at http://localhost:8080 for monitoring and configuration
4. Web UI provides real-time status, logs, and schedule management
5. Daemon can be stopped with garminsync daemon --stop or through web UI

---

File Structure

/garminsync
├── garminsync/              # Main application package
│   ├── __init__.py         # Empty package file
│   ├── cli.py             # Typer CLI commands and main entrypoint
│   ├── config.py          # Configuration and environment variable loading
│   ├── database.py        # SQLAlchemy models and database operations
│   ├── garmin.py          # Garmin Connect client wrapper with robust download logic
│   ├── daemon.py          # Daemon mode implementation with APScheduler
│   ├── web/               # Web UI components
│   │   ├── __init__.py
│   │   ├── app.py         # FastAPI application setup
│   │   ├── routes.py      # API endpoints for web UI
│   │   ├── static/        # CSS, JavaScript, images
│   │   │   ├── style.css
│   │   │   └── app.js
│   │   └── templates/     # Jinja2 HTML templates
│   │       ├── base.html
│   │       ├── dashboard.html
│   │       └── config.html
│   └── utils.py           # Shared utilities and helpers
├── data/                    # Directory for downloaded .fit files and SQLite DB
├── .env                     # Stores GARMIN_EMAIL/GARMIN_PASSWORD (gitignored)
├── .gitignore              # Excludes .env file
├── Dockerfile              # Production-ready container configuration
├── Design.md               # This design document
└── requirements.txt        # Pinned Python dependencies (updated)

---

Technical Implementation Details

Architecture

• CLI Framework: Uses Typer with proper type hints and validation
• Module Separation: Clear separation between CLI commands, database operations, and Garmin API interactions
• Error Handling: Comprehensive exception handling with user-friendly error messages
• Session Management: Proper SQLAlchemy session management with cleanup

Authentication & Configuration

• Credentials loaded via python-dotenv from environment variables
• Configuration validation ensures required credentials are present
• Garmin client handles authentication automatically with session persistence

Database Operations

• SQLite database with SQLAlchemy ORM for type safety
• Database initialization creates tables automatically
• Sync functionality reconciles local database with Garmin Connect activities
• Proper transaction management with rollback on errors

File Management

• Files named with pattern: activity_{activity_id}_{timestamp}.fit
• Timestamp sanitized for filesystem compatibility (colons and spaces replaced)
• Downloads saved to configurable data directory
• Database tracks both download status and file paths

API Integration

• Rate Limiting: 2-second delays between API requests to respect Garmin's servers
• Robust Downloads: Multiple fallback methods for downloading FIT files:
  1. Default download method
  2. Explicit 'fit' format parameter
  3. Alternative parameter names and formats
  4. Graceful fallback with detailed error reporting
• Activity Fetching: Configurable batch sizes (currently 1000 activities per sync)

User Experience

• Progress Indicators: tqdm progress bars for all long-running operations
• Informative Output: Clear status messages and operation summaries
• Input Validation: Prevents invalid command combinations
• Exit Codes: Proper exit codes for script integration

---

Development Status ✅

✅ Completed Features

Phase 1: Core Infrastructure

• Dockerfile: Production-ready Python 3.10 container with proper layer caching
• Environment Configuration: python-dotenv integration with validation
• CLI Framework: Complete Typer implementation with type hints and help text
• Garmin Integration: Robust python-garminconnect wrapper with authentication

Phase 2: Activity Listing

• Database Schema: SQLAlchemy models with proper relationships
• Database Operations: Session management, initialization, and sync functionality
• List Commands: All filter options (--all, --missing, --downloaded) implemented
• Progress Display: tqdm integration for user feedback during operations

Phase 3: Download Pipeline

• FIT File Downloads: Multi-method download approach with fallback strategies
• Idempotent Operations: Prevents re-downloading existing files
• Database Updates: Proper status tracking and file path storage
• File Management: Safe filename generation and directory creation

Phase 4: Polish

• Progress Bars: Comprehensive tqdm implementation across all operations
• Error Handling: Graceful error handling with informative messages
• Container Optimization: Efficient Docker build with proper dependency management

🚧 New Features Implementation Guide

Feature 1: Offline Mode

Implementation Steps:

1. CLI Enhancement (cli.py):

   @app.command("list")
   def list_activities(
       all_activities: bool = False,
       missing: bool = False,
       downloaded: bool = False,
       offline: Annotated[bool, typer.Option("--offline", help="Work offline without syncing")] = False
   ):
       if not offline:
           # Existing sync logic
           sync_database(client)
       else:
           typer.echo("Working in offline mode - using cached data")
   
       # Rest of listing logic remains the same
   

2. Database Enhancements (database.py):

   • Add last_sync column to Activity table
   • Add utility functions for offline status checking

   def get_offline_stats():
       """Return statistics about cached data without API calls"""
       session = get_session()
       total = session.query(Activity).count()
       downloaded = session.query(Activity).filter_by(downloaded=True).count()
       missing = total - downloaded
       last_sync = session.query(Activity).order_by(Activity.last_sync.desc()).first()
       return {
           'total': total,
           'downloaded': downloaded, 
           'missing': missing,
           'last_sync': last_sync.last_sync if last_sync else 'Never'
       }
   

Feature 2: Daemon Mode

Implementation Steps:

1. New Daemon Module (daemon.py):

   from apscheduler.schedulers.background import BackgroundScheduler
   from apscheduler.triggers.cron import CronTrigger
   import signal
   import sys
   import time
   import threading
   from datetime import datetime
   
   class GarminSyncDaemon:
       def __init__(self):
           self.scheduler = BackgroundScheduler()
           self.running = False
           self.web_server = None
   
       def start(self, web_port=8080):
           """Start daemon with scheduler and web UI"""
           # Load configuration from database
           config = self.load_config()
   
           # Setup scheduled job
           if config.enabled:
               self.scheduler.add_job(
                   func=self.sync_and_download,
                   trigger=CronTrigger.from_crontab(config.schedule_cron),
                   id='sync_job',
                   replace_existing=True
               )
   
           # Start scheduler
           self.scheduler.start()
           self.running = True
   
           # Start web UI in separate thread
           self.start_web_ui(web_port)
   
           # Setup signal handlers for graceful shutdown
           signal.signal(signal.SIGINT, self.signal_handler)
           signal.signal(signal.SIGTERM, self.signal_handler)
   
           print(f"Daemon started. Web UI available at http://localhost:{web_port}")
   
           # Keep daemon running
           try:
               while self.running:
                   time.sleep(1)
           except KeyboardInterrupt:
               self.stop()
   
       def sync_and_download(self):
           """Scheduled job function"""
           try:
               self.log_operation("sync", "started")
   
               # Perform sync and download
               from .garmin import GarminClient
               from .database import sync_database
   
               client = GarminClient()
               activities_before = self.count_missing()
   
               sync_database(client)
   
               # Download missing activities
               downloaded_count = self.download_missing_activities(client)
   
               self.log_operation("sync", "success", 
                   f"Downloaded {downloaded_count} new activities")
   
           except Exception as e:
               self.log_operation("sync", "error", str(e))
   
       def load_config(self):
           """Load daemon configuration from database"""
           session = get_session()
           config = session.query(DaemonConfig).first()
           if not config:
               # Create default configuration
               config = DaemonConfig()
               session.add(config)
               session.commit()
           session.close()
           return config
   

2. CLI Integration (cli.py):

   @app.command("daemon")
   def daemon_mode(
       start: Annotated[bool, typer.Option("--start", help="Start daemon")] = False,
       stop: Annotated[bool, typer.Option("--stop", help="Stop daemon")] = False,
       status: Annotated[bool, typer.Option("--status", help="Show daemon status")] = False,
       port: Annotated[int, typer.Option("--port", help="Web UI port")] = 8080
   ):
       """Daemon mode operations"""
       from .daemon import GarminSyncDaemon
   
       if start:
           daemon = GarminSyncDaemon()
           daemon.start(web_port=port)
       elif stop:
           # Implementation for stopping daemon (PID file or signal)
           pass
       elif status:
           # Show current daemon status
           pass
   

Feature 3: Web UI

Implementation Steps:

1. FastAPI Application (web/app.py):

   from fastapi import FastAPI, Request
   from fastapi.staticfiles import StaticFiles
   from fastapi.templating import Jinja2Templates
   from .routes import router
   
   app = FastAPI(title="GarminSync Dashboard")
   
   # Mount static files and templates
   app.mount("/static", StaticFiles(directory="garminsync/web/static"), name="static")
   templates = Jinja2Templates(directory="garminsync/web/templates")
   
   # Include API routes
   app.include_router(router)
   
   @app.get("/")
   async def dashboard(request: Request):
       # Get current statistics
       from ..database import get_offline_stats
       stats = get_offline_stats()
   
       return templates.TemplateResponse("dashboard.html", {
           "request": request,
           "stats": stats
       })
   

2. API Routes (web/routes.py):

   from fastapi import APIRouter, HTTPException
   from pydantic import BaseModel
   from ..database import get_session, DaemonConfig, SyncLog
   
   router = APIRouter(prefix="/api")
   
   class ScheduleConfig(BaseModel):
       enabled: bool
       cron_schedule: str
   
   @router.get("/status")
   async def get_status():
       """Get current daemon status"""
       session = get_session()
       config = session.query(DaemonConfig).first()
   
       # Get recent logs
       logs = session.query(SyncLog).order_by(SyncLog.timestamp.desc()).limit(10).all()
   
       return {
           "daemon": {
               "running": config.status == "running" if config else False,
               "next_run": config.next_run if config else None,
               "schedule": config.schedule_cron if config else None
           },
           "recent_logs": [
               {
                   "timestamp": log.timestamp,
                   "operation": log.operation,
                   "status": log.status,
                   "message": log.message
               } for log in logs
           ]
       }
   
   @router.post("/schedule")
   async def update_schedule(config: ScheduleConfig):
       """Update daemon schedule configuration"""
       session = get_session()
       daemon_config = session.query(DaemonConfig).first()
   
       if not daemon_config:
           daemon_config = DaemonConfig()
           session.add(daemon_config)
   
       daemon_config.enabled = config.enabled
       daemon_config.schedule_cron = config.cron_schedule
       session.commit()
   
       return {"message": "Configuration updated successfully"}
   
   @router.post("/sync/trigger")
   async def trigger_sync():
       """Manually trigger a sync operation"""
       # Implementation to trigger immediate sync
       pass
   

3. HTML Templates (web/templates/dashboard.html):

   {% extends "base.html" %}
   
   {% block content %}
   <div class="container">
       <h1>GarminSync Dashboard</h1>
   
       <div class="row">
           <div class="col-md-4">
               <div class="card">
                   <div class="card-header">Statistics</div>
                   <div class="card-body">
                       <p>Total Activities: {{ stats.total }}</p>
                       <p>Downloaded: {{ stats.downloaded }}</p>
                       <p>Missing: {{ stats.missing }}</p>
                       <p>Last Sync: {{ stats.last_sync }}</p>
                   </div>
               </div>
           </div>
   
           <div class="col-md-4">
               <div class="card">
                   <div class="card-header">Daemon Status</div>
                   <div class="card-body" id="daemon-status">
                       <!-- Populated by JavaScript -->
                   </div>
               </div>
           </div>
   
           <div class="col-md-4">
               <div class="card">
                   <div class="card-header">Quick Actions</div>
                   <div class="card-body">
                       <button class="btn btn-primary" onclick="triggerSync()">
                           Sync Now
                       </button>
                       <button class="btn btn-secondary" onclick="toggleDaemon()">
                           Toggle Daemon
                       </button>
                   </div>
               </div>
           </div>
       </div>
   
       <div class="row mt-4">
           <div class="col-12">
               <div class="card">
                   <div class="card-header">Recent Activity</div>
                   <div class="card-body" id="recent-logs">
                       <!-- Populated by JavaScript -->
                   </div>
               </div>
           </div>
       </div>
   
       <div class="row mt-4">
           <div class="col-12">
               <div class="card">
                   <div class="card-header">Schedule Configuration</div>
                   <div class="card-body">
                       <form id="schedule-form">
                           <div class="form-group">
                               <label for="schedule-enabled">Enable Scheduled Sync</label>
                               <input type="checkbox" id="schedule-enabled">
                           </div>
                           <div class="form-group">
                               <label for="cron-schedule">Cron Schedule</label>
                               <input type="text" class="form-control" id="cron-schedule" 
                                      placeholder="0 */6 * * *" title="Every 6 hours">
                           </div>
                           <button type="submit" class="btn btn-primary">
                               Update Schedule
                           </button>
                       </form>
                   </div>
               </div>
           </div>
       </div>
   </div>
   {% endblock %}
   

4. JavaScript for Interactivity (web/static/app.js):

   // Auto-refresh dashboard data
   setInterval(updateStatus, 30000); // Every 30 seconds
   
   async function updateStatus() {
       try {
           const response = await fetch('/api/status');
           const data = await response.json();
   
           // Update daemon status
           document.getElementById('daemon-status').innerHTML = `
               <p>Status: <span class="badge ${data.daemon.running ? 'badge-success' : 'badge-danger'}">
                   ${data.daemon.running ? 'Running' : 'Stopped'}
               </span></p>
               <p>Next Run: ${data.daemon.next_run || 'Not scheduled'}</p>
               <p>Schedule: ${data.daemon.schedule || 'Not configured'}</p>
           `;
   
           // Update recent logs
           const logsHtml = data.recent_logs.map(log => `
               <div class="log-entry">
                   <small class="text-muted">${log.timestamp}</small>
                   <span class="badge badge-${log.status === 'success' ? 'success' : 'danger'}">
                       ${log.status}
                   </span>
                   ${log.operation}: ${log.message || ''}
               </div>
           `).join('');
   
           document.getElementById('recent-logs').innerHTML = logsHtml;
   
       } catch (error) {
           console.error('Failed to update status:', error);
       }
   }
   
   async function triggerSync() {
       try {
           await fetch('/api/sync/trigger', { method: 'POST' });
           alert('Sync triggered successfully');
           updateStatus();
       } catch (error) {
           alert('Failed to trigger sync');
       }
   }
   
   // Initialize on page load
   document.addEventListener('DOMContentLoaded', updateStatus);
   

Updated Requirements (requirements.txt):

typer==0.9.0
click==8.1.7
python-dotenv==1.0.0
garminconnect==0.2.28
sqlalchemy==2.0.23
tqdm==4.66.1
fastapi==0.104.1
uvicorn[standard]==0.24.0
apscheduler==3.10.4
pydantic==2.5.0
jinja2==3.1.2
python-multipart==0.0.6
aiofiles==23.2.1

Docker Updates:

# Expose web UI port
EXPOSE 8080

# Update entrypoint to support daemon mode
ENTRYPOINT ["python", "-m", "garminsync.cli"]
CMD ["--help"]

Usage Examples:

Offline Mode:

# List missing activities without network calls
docker run --env-file .env -v $(pwd)/data:/app/data garminsync list --missing --offline

Daemon Mode:

# Start daemon with web UI on port 8080
docker run -d --env-file .env -v $(pwd)/data:/app/data -p 8080:8080 garminsync daemon --start

# Access web UI at http://localhost:8080

---

Docker Usage

Build the Container

docker build -t garminsync .

Run with Environment File

docker run -it --env-file .env -v $(pwd)/data:/app/data garminsync --help

Example Commands

# List all activities
docker run -it --env-file .env -v $(pwd)/data:/app/data garminsync list --all

# Download missing activities
docker run -it --env-file .env -v $(pwd)/data:/app/data garminsync download --missing

---

Environment Setup

Create a .env file in the project root:

GARMIN_EMAIL=your_email@example.com
GARMIN_PASSWORD=your_password

---

Key Implementation Highlights

Robust Download Logic

The garmin.py module implements a sophisticated download strategy that tries multiple methods to handle variations in the Garmin Connect API:

methods_to_try = [
    lambda: self.client.download_activity(activity_id),
    lambda: self.client.download_activity(activity_id, fmt='fit'),
    lambda: self.client.download_activity(activity_id, format='fit'),
    # ... additional fallback methods
]

Database Synchronization

The sync process efficiently updates the local database with new activities from Garmin Connect:

def sync_database(garmin_client):
    """Sync local database with Garmin Connect activities"""
    activities = garmin_client.get_activities(0, 1000)
    for activity in activities:
        # Only add new activities, preserve existing download status
        existing = session.query(Activity).filter_by(activity_id=activity_id).first()
        if not existing:
            new_activity = Activity(...)
            session.add(new_activity)

CLI Integration

Clean separation between CLI interface and business logic with proper type annotations:

def list_activities(
    all_activities: Annotated[bool, typer.Option("--all", help="List all activities")] = False,
    missing: Annotated[bool, typer.Option("--missing", help="List missing activities")] = False,
    downloaded: Annotated[bool, typer.Option("--downloaded", help="List downloaded activities")] = False
):

---

Documentation 📚

Here are links to the official documentation for the key libraries used:

• Garmin API: python-garminconnect
• CLI Framework: Typer
• Environment Variables: python-dotenv
• Database ORM: SQLAlchemy
• Progress Bars: tqdm

---

Performance Considerations

• Rate Limiting: 2-second delays between API requests prevent server overload
• Batch Processing: Fetches up to 1000 activities per sync operation
• Efficient Queries: Database queries optimized for filtering operations
• Memory Management: Proper session cleanup and resource management
• Docker Optimization: Layer caching and minimal base image for faster builds