checkpoint 2

Dockerfile

@@ -19,6 +19,7 @@ RUN pip install --upgrade pip && \
 # Copy application code
 COPY garminsync/ ./garminsync/
 COPY migrations/ ./migrations/
+COPY migrations/alembic.ini ./alembic.ini
 COPY tests/ ./tests/
 COPY entrypoint.sh .
 COPY patches/ ./patches/

cyclingpower.md

@@ -0,0 +1,221 @@
+# Cycling FIT Analysis Implementation Plan
+
+## Overview
+Extend the existing GarminSync FIT parser to calculate cycling-specific metrics including power estimation and singlespeed gear ratio analysis for activities without native power data.
+
+## Phase 1: Core Infrastructure Setup
+
+### 1.1 Database Schema Extensions
+**File: `garminsync/database.py`**
+- Extend existing `PowerAnalysis` table with cycling-specific fields:
+  ```python
+  # Add to PowerAnalysis class:
+  peak_power_1s = Column(Float, nullable=True)
+  peak_power_5s = Column(Float, nullable=True) 
+  peak_power_20s = Column(Float, nullable=True)
+  peak_power_300s = Column(Float, nullable=True)
+  normalized_power = Column(Float, nullable=True)
+  intensity_factor = Column(Float, nullable=True)
+  training_stress_score = Column(Float, nullable=True)
+  ```
+
+- Extend existing `GearingAnalysis` table:
+  ```python
+  # Add to GearingAnalysis class:
+  estimated_chainring_teeth = Column(Integer, nullable=True)
+  estimated_cassette_teeth = Column(Integer, nullable=True)
+  gear_ratio = Column(Float, nullable=True)
+  gear_inches = Column(Float, nullable=True)
+  development_meters = Column(Float, nullable=True)
+  confidence_score = Column(Float, nullable=True)
+  analysis_method = Column(String, default="singlespeed_estimation")
+  ```
+
+### 1.2 Enhanced FIT Parser
+**File: `garminsync/fit_processor/parser.py`**
+- Extend `FITParser` to extract cycling-specific data points:
+  ```python
+  def _extract_cycling_data(self, message):
+      """Extract cycling-specific metrics from FIT records"""
+      # GPS coordinates for elevation/gradient
+      # Speed and cadence for gear analysis
+      # Power data (if available) for validation
+      # Temperature for air density calculations
+  ```
+
+## Phase 2: Power Estimation Engine
+
+### 2.1 Physics-Based Power Calculator
+**New file: `garminsync/fit_processor/power_estimator.py`**
+
+**Key Components:**
+- **Environmental factors**: Air density, wind resistance, temperature
+- **Bike specifications**: Weight (22 lbs = 10 kg), aerodynamic drag coefficient
+- **Rider assumptions**: Weight (75 kg default), position (road bike)
+- **Terrain analysis**: Gradient calculation from GPS elevation data
+
+**Core Algorithm:**
+```python
+class PowerEstimator:
+    def __init__(self):
+        self.bike_weight_kg = 10.0  # 22 lbs
+        self.rider_weight_kg = 75.0  # Default assumption
+        self.drag_coefficient = 0.88  # Road bike
+        self.frontal_area_m2 = 0.4  # Typical road cycling position
+        self.rolling_resistance = 0.004  # Road tires
+        self.drivetrain_efficiency = 0.97
+        self.air_density = 1.225  # kg/m³ at sea level, 20°C
+    
+    def calculate_power(self, speed_ms, gradient_percent, 
+                       air_temp_c=20, altitude_m=0):
+        """Calculate estimated power using physics model"""
+        # Power = (Rolling + Gravity + Aerodynamic + Kinetic) / Efficiency
+```
+
+**Power Components:**
+1. **Rolling resistance**: `P_roll = Crr × (m_bike + m_rider) × g × cos(θ) × v`
+2. **Gravitational**: `P_grav = (m_bike + m_rider) × g × sin(θ) × v`
+3. **Aerodynamic**: `P_aero = 0.5 × ρ × Cd × A × v³`
+4. **Acceleration**: `P_accel = (m_bike + m_rider) × a × v`
+
+### 2.2 Peak Power Analysis
+**Methods:**
+- 1-second, 5-second, 20-second, 5-minute peak power windows
+- Normalized Power (NP) calculation using 30-second rolling average
+- Training Stress Score (TSS) estimation based on NP and ride duration
+
+## Phase 3: Singlespeed Gear Ratio Analysis
+
+### 3.1 Gear Ratio Calculator
+**New file: `garminsync/fit_processor/gear_analyzer.py`**
+
+**Strategy:**
+- Analyze flat terrain segments (gradient < 3%)
+- Use speed/cadence relationship to determine gear ratio
+- Test against common singlespeed ratios for 38t and 46t chainrings
+- Calculate confidence scores based on data consistency
+
+**Core Algorithm:**
+```python
+class SinglespeedAnalyzer:
+    def __init__(self):
+        self.chainring_options = [38, 46]  # teeth
+        self.common_cogs = list(range(11, 28))  # 11t to 27t rear cogs
+        self.wheel_circumference_m = 2.096  # 700x25c tire
+    
+    def analyze_gear_ratio(self, speed_data, cadence_data, gradient_data):
+        """Determine most likely singlespeed gear ratio"""
+        # Filter for flat terrain segments
+        # Calculate gear ratio from speed/cadence
+        # Match against common ratios
+        # Return best fit with confidence score
+```
+
+**Gear Metrics:**
+- **Gear ratio**: Chainring teeth ÷ Cog teeth
+- **Gear inches**: Gear ratio × wheel diameter (inches)
+- **Development**: Distance traveled per pedal revolution (meters)
+
+### 3.2 Analysis Methodology
+1. **Segment filtering**: Identify flat terrain (gradient < 3%, speed > 15 km/h)
+2. **Ratio calculation**: `gear_ratio = (speed_ms × 60) ÷ (cadence_rpm × wheel_circumference_m)`
+3. **Ratio matching**: Compare calculated ratios against theoretical singlespeed options
+4. **Confidence scoring**: Based on data consistency and segment duration
+
+## Phase 4: Integration with Existing System
+
+### 4.1 FIT Processing Workflow Enhancement
+**File: `garminsync/fit_processor/analyzer.py`**
+- Integrate power estimation and gear analysis into existing analysis workflow
+- Add cycling-specific analysis triggers (detect cycling activities)
+- Store results in database using existing schema
+
+### 4.2 Database Population
+**Migration strategy:**
+- Extend existing migration system to handle new fields
+- Process existing FIT files retroactively
+- Add processing status tracking for cycling analysis
+
+### 4.3 CLI Integration
+**File: `garminsync/cli.py`**
+- Add new command: `garminsync analyze --cycling --activity-id <id>`
+- Add batch processing: `garminsync analyze --cycling --missing`
+- Add reporting: `garminsync report --power-analysis --gear-analysis`
+
+## Phase 5: Validation and Testing
+
+### 5.1 Test Data Requirements
+- FIT files with known power data for validation
+- Various singlespeed configurations for gear ratio testing
+- Different terrain types (flat, climbing, mixed)
+
+### 5.2 Validation Methodology
+- Compare estimated vs. actual power (where available)
+- Validate gear ratio estimates against known bike configurations
+- Test edge cases (very low/high cadence, extreme gradients)
+
+### 5.3 Performance Optimization
+- Efficient gradient calculation from GPS data
+- Optimize power calculation loops for large datasets
+- Cache intermediate calculations
+
+## Phase 6: Advanced Features (Future)
+
+### 6.1 Environmental Corrections
+- Wind speed/direction integration
+- Barometric pressure for accurate altitude
+- Temperature-based air density adjustments
+
+### 6.2 Machine Learning Enhancement
+- Train models on validated power data
+- Improve gear ratio detection accuracy
+- Personalized power estimation based on rider history
+
+### 6.3 Comparative Analysis
+- Compare estimated metrics across rides
+- Trend analysis for fitness progression
+- Gear ratio optimization recommendations
+
+## Implementation Priority
+
+**High Priority:**
+1. Database schema extensions
+2. Basic power estimation using physics model
+3. Singlespeed gear ratio analysis for flat terrain
+4. Integration with existing FIT processing pipeline
+
+**Medium Priority:**
+1. Peak power analysis (1s, 5s, 20s, 5min)
+2. Normalized Power and TSS calculations
+3. Advanced gear analysis with confidence scoring
+4. CLI commands for analysis and reporting
+
+**Low Priority:**
+1. Environmental corrections (wind, pressure)
+2. Machine learning enhancements
+3. Advanced comparative analysis features
+4. Web UI integration for visualizing results
+
+## Success Criteria
+
+1. **Power Estimation**: Within ±10% of actual power data (where available for validation)
+2. **Gear Ratio Detection**: Correctly identify gear ratios within ±1 tooth accuracy
+3. **Processing Speed**: Analyze typical FIT file (1-hour ride) in <5 seconds
+4. **Data Coverage**: Successfully analyze 90%+ of cycling FIT files
+5. **Integration**: Seamlessly integrate with existing GarminSync workflow
+
+## File Structure Summary
+
+```
+garminsync/
+├── fit_processor/
+│   ├── parser.py (enhanced)
+│   ├── analyzer.py (enhanced) 
+│   ├── power_estimator.py (new)
+│   └── gear_analyzer.py (new)
+├── database.py (enhanced)
+├── cli.py (enhanced)
+└── migrate_cycling_analysis.py (new)
+```
+
+This plan provides a comprehensive roadmap for implementing cycling-specific FIT analysis while building on the existing GarminSync infrastructure and maintaining compatibility with current functionality.

entrypoint.sh

@@ -1,16 +1,31 @@
 #!/bin/bash
 
-# Run database migrations
-echo "Running database migrations..."
-export ALEMBIC_CONFIG=${ALEMBIC_CONFIG:-./migrations/alembic.ini}
-export ALEMBIC_SCRIPT_LOCATION=${ALEMBIC_SCRIPT_LOCATION:-./migrations/versions}
-alembic upgrade head
-if [ $? -ne 0 ]; then
-    echo "Migration failed!" >&2
-    exit 1
+# Conditionally run database migrations
+if [ "${RUN_MIGRATIONS:-1}" = "1" ]; then
+    echo "$(date) - Starting database migrations..."
+    echo "ALEMBIC_CONFIG: ${ALEMBIC_CONFIG:-/app/migrations/alembic.ini}"
+    echo "ALEMBIC_SCRIPT_LOCATION: ${ALEMBIC_SCRIPT_LOCATION:-/app/migrations/versions}"
+    
+    # Run migrations with timing
+    start_time=$(date +%s)
+    export ALEMBIC_CONFIG=${ALEMBIC_CONFIG:-/app/migrations/alembic.ini}
+    export ALEMBIC_SCRIPT_LOCATION=${ALEMBIC_SCRIPT_LOCATION:-/app/migrations/versions}
+    alembic upgrade head
+    migration_status=$?
+    end_time=$(date +%s)
+    duration=$((end_time - start_time))
+    
+    if [ $migration_status -ne 0 ]; then
+        echo "$(date) - Migration failed after ${duration} seconds!" >&2
+        exit 1
+    else
+        echo "$(date) - Migrations completed successfully in ${duration} seconds"
+    fi
+else
+    echo "$(date) - Skipping database migrations (RUN_MIGRATIONS=${RUN_MIGRATIONS})"
 fi
 
 # Start the application
-echo "Starting application..."
+echo "$(date) - Starting application..."
 exec python -m garminsync.cli daemon --start --port 8888
 sleep infinity

garminsync/activity_parser.py

@@ -0,0 +1,130 @@
+import os
+import gzip
+import fitdecode
+import xml.etree.ElementTree as ET
+from datetime import datetime
+
+def detect_file_type(file_path):
+    """Detect file format (FIT, XML, or unknown)"""
+    try:
+        with open(file_path, 'rb') as f:
+            header = f.read(128)
+            if b'<?xml' in header[:20]:
+                return 'xml'
+            if len(header) >= 8 and header[4:8] == b'.FIT':
+                return 'fit'
+            if (len(header) >= 8 and 
+                (header[0:4] == b'.FIT' or 
+                 header[4:8] == b'FIT.' or 
+                 header[8:12] == b'.FIT')):
+                return 'fit'
+            return 'unknown'
+    except Exception as e:
+        return 'error'
+
+def parse_xml_file(file_path):
+    """Parse XML (TCX) file to extract activity metrics"""
+    try:
+        tree = ET.parse(file_path)
+        root = tree.getroot()
+        namespaces = {'ns': 'http://www.garmin.com/xmlschemas/TrainingCenterDatabase/v2'}
+        
+        sport = root.find('.//ns:Activity', namespaces).get('Sport', 'other')
+        distance = root.find('.//ns:DistanceMeters', namespaces)
+        distance = float(distance.text) if distance is not None else None
+        duration = root.find('.//ns:TotalTimeSeconds', namespaces)
+        duration = float(duration.text) if duration is not None else None
+        calories = root.find('.//ns:Calories', namespaces)
+        calories = int(calories.text) if calories is not None else None
+        
+        hr_values = []
+        for hr in root.findall('.//ns:HeartRateBpm/ns:Value', namespaces):
+            try:
+                hr_values.append(int(hr.text))
+            except:
+                continue
+        max_hr = max(hr_values) if hr_values else None
+        
+        return {
+            "activityType": {"typeKey": sport},
+            "summaryDTO": {
+                "duration": duration,
+                "distance": distance,
+                "maxHR": max_hr,
+                "avgPower": None,
+                "calories": calories
+            }
+        }
+    except Exception:
+        return None
+
+def parse_fit_file(file_path):
+    """Parse FIT file to extract activity metrics"""
+    metrics = {}
+    try:
+        with open(file_path, 'rb') as f:
+            magic = f.read(2)
+            f.seek(0)
+            is_gzipped = magic == b'\x1f\x8b'
+        
+        if is_gzipped:
+            with gzip.open(file_path, 'rb') as gz_file:
+                from io import BytesIO
+                with BytesIO(gz_file.read()) as fit_data:
+                    fit = fitdecode.FitReader(fit_data)
+                    for frame in fit:
+                        if frame.frame_type == fitdecode.FrameType.DATA and frame.name == 'session':
+                            metrics = {
+                                "sport": frame.get_value("sport"),
+                                "total_timer_time": frame.get_value("total_timer_time"),
+                                "total_distance": frame.get_value("total_distance"),
+                                "max_heart_rate": frame.get_value("max_heart_rate"),
+                                "avg_power": frame.get_value("avg_power"),
+                                "total_calories": frame.get_value("total_calories")
+                            }
+                            break
+        else:
+            with fitdecode.FitReader(file_path) as fit:
+                for frame in fit:
+                    if frame.frame_type == fitdecode.FrameType.DATA and frame.name == 'session':
+                        metrics = {
+                            "sport": frame.get_value("sport"),
+                            "total_timer_time": frame.get_value("total_timer_time"),
+                            "total_distance": frame.get_value("total_distance"),
+                            "max_heart_rate": frame.get_value("max_heart_rate"),
+                            "avg_power": frame.get_value("avg_power"),
+                            "total_calories": frame.get_value("total_calories")
+                        }
+                        break
+    
+        return {
+            "activityType": {"typeKey": metrics.get("sport", "other")},
+            "summaryDTO": {
+                "duration": metrics.get("total_timer_time"),
+                "distance": metrics.get("total_distance"),
+                "maxHR": metrics.get("max_heart_rate"),
+                "avgPower": metrics.get("avg_power"),
+                "calories": metrics.get("total_calories")
+            }
+        }
+    except Exception:
+        return None
+
+def get_activity_metrics(activity, client=None):
+    """
+    Get activity metrics from local file or Garmin API
+    Returns parsed metrics or None
+    """
+    metrics = None
+    if activity.filename and os.path.exists(activity.filename):
+        file_type = detect_file_type(activity.filename)
+        if file_type == 'fit':
+            metrics = parse_fit_file(activity.filename)
+        elif file_type == 'xml':
+            metrics = parse_xml_file(activity.filename)
+    if not metrics and client:
+        try:
+            metrics = client.get_activity_details(activity.activity_id)
+        except Exception:
+            pass
+    return metrics

garminsync/cli.py

@@ -173,13 +173,20 @@ def daemon_mode(
         bool, typer.Option("--status", help="Show daemon status")
     ] = False,
     port: Annotated[int, typer.Option("--port", help="Web UI port")] = 8080,
+    run_migrations: Annotated[
+        bool, 
+        typer.Option(
+            "--run-migrations/--skip-migrations", 
+            help="Run database migrations on startup (default: run)"
+        )
+    ] = True,
 ):
     """Daemon mode operations"""
     from .daemon import GarminSyncDaemon
 
     if start:
         daemon = GarminSyncDaemon()
-        daemon.start(web_port=port)
+        daemon.start(web_port=port, run_migrations=run_migrations)
     elif stop:
         # Implementation for stopping daemon (PID file or signal)
         typer.echo("Stopping daemon...")

garminsync/daemon.py

@@ -10,6 +10,7 @@ from apscheduler.triggers.cron import CronTrigger
 from .database import Activity, DaemonConfig, SyncLog, get_session
 from .garmin import GarminClient
 from .utils import logger
+from .activity_parser import get_activity_metrics
 
 
 class GarminSyncDaemon:
@@ -18,8 +19,17 @@ class GarminSyncDaemon:
         self.running = False
         self.web_server = None
 
-    def start(self, web_port=8888):
-        """Start daemon with scheduler and web UI"""
+    def start(self, web_port=8888, run_migrations=True):
+        """Start daemon with scheduler and web UI
+        :param web_port: Port for the web UI
+        :param run_migrations: Whether to run database migrations on startup
+        """
+        # Set migration flag for entrypoint
+        if run_migrations:
+            os.environ['RUN_MIGRATIONS'] = "1"
+        else:
+            os.environ['RUN_MIGRATIONS'] = "0"
+            
         try:
             # Load configuration from database
             config_data = self.load_config()
@@ -105,26 +115,38 @@ class GarminSyncDaemon:
 
             for activity in missing_activities:
                 try:
-                    # Use the correct method name
+                    # Download FIT file
                     fit_data = client.download_activity_fit(activity.activity_id)
-
-                    # Save the file
+                    
+                    # Save to file
                     import os
                     from pathlib import Path
-
                     data_dir = Path(os.getenv("DATA_DIR", "data"))
                     data_dir.mkdir(parents=True, exist_ok=True)
-
                     timestamp = activity.start_time.replace(":", "-").replace(" ", "_")
                     filename = f"activity_{activity.activity_id}_{timestamp}.fit"
                     filepath = data_dir / filename
-
+                    
                     with open(filepath, "wb") as f:
                         f.write(fit_data)
-
+                    
+                    # Update activity record
                     activity.filename = str(filepath)
                     activity.downloaded = True
                     activity.last_sync = datetime.now().isoformat()
+                    
+                    # Get metrics immediately after download
+                    metrics = get_activity_metrics(activity, client)
+                    if metrics:
+                        # Update metrics if available
+                        activity.activity_type = metrics.get("activityType", {}).get("typeKey")
+                        activity.duration = int(float(metrics.get("summaryDTO", {}).get("duration", 0)))
+                        activity.distance = float(metrics.get("summaryDTO", {}).get("distance", 0))
+                        activity.max_heart_rate = int(float(metrics.get("summaryDTO", {}).get("maxHR", 0)))
+                        activity.avg_power = float(metrics.get("summaryDTO", {}).get("avgPower", 0))
+                        activity.calories = int(float(metrics.get("summaryDTO", {}).get("calories", 0)))
+                    
+                    session.commit()
                     downloaded_count += 1
                     session.commit()
 
@@ -135,7 +157,8 @@ class GarminSyncDaemon:
                     session.rollback()
 
             self.log_operation(
-                "sync", "success", f"Downloaded {downloaded_count} new activities"
+                "sync", "success", 
+                f"Downloaded {downloaded_count} new activities and updated metrics"
             )
 
             # Update last run time

garminsync/database.py

@@ -120,6 +120,8 @@ def get_session():
     return Session()
 
 
+from garminsync.activity_parser import get_activity_metrics
+
 def sync_database(garmin_client):
     """Sync local database with Garmin Connect activities.
 
@@ -134,36 +136,70 @@ def sync_database(garmin_client):
             print("No activities returned from Garmin API")
             return
 
-        for activity in activities:
-            # Check if activity is a dictionary and has required fields
-            if not isinstance(activity, dict):
-                print(f"Invalid activity data: {activity}")
+        for activity_data in activities:
+            if not isinstance(activity_data, dict):
+                print(f"Invalid activity data: {activity_data}")
                 continue
 
-            # Safely access dictionary keys
-            activity_id = activity.get("activityId")
-            start_time = activity.get("startTimeLocal")
-            avg_heart_rate = activity.get("averageHR", None)
-            calories = activity.get("calories", None)
-
+            activity_id = activity_data.get("activityId")
+            start_time = activity_data.get("startTimeLocal")
+            
             if not activity_id or not start_time:
-                print(f"Missing required fields in activity: {activity}")
+                print(f"Missing required fields in activity: {activity_data}")
                 continue
 
-            existing = (
-                session.query(Activity).filter_by(activity_id=activity_id).first()
-            )
+            existing = session.query(Activity).filter_by(activity_id=activity_id).first()
+            
+            # Create or update basic activity info
             if not existing:
-                new_activity = Activity(
+                activity = Activity(
                     activity_id=activity_id,
                     start_time=start_time,
-                    avg_heart_rate=avg_heart_rate,
-                    calories=calories,
                     downloaded=False,
                     created_at=datetime.now().isoformat(),
                     last_sync=datetime.now().isoformat(),
                 )
-                session.add(new_activity)
+                session.add(activity)
+                session.flush()  # Assign ID
+            else:
+                activity = existing
+            
+            # Update metrics using shared parser
+            metrics = get_activity_metrics(activity, garmin_client)
+            if metrics:
+                activity.activity_type = metrics.get("activityType", {}).get("typeKey")
+                
+                # Extract duration in seconds
+                duration = metrics.get("summaryDTO", {}).get("duration")
+                if duration is not None:
+                    activity.duration = int(float(duration))
+                
+                # Extract distance in meters
+                distance = metrics.get("summaryDTO", {}).get("distance")
+                if distance is not None:
+                    activity.distance = float(distance)
+                
+                # Extract heart rates
+                max_hr = metrics.get("summaryDTO", {}).get("maxHR")
+                if max_hr is not None:
+                    activity.max_heart_rate = int(float(max_hr))
+                
+                avg_hr = metrics.get("summaryDTO", {}).get("avgHR", None) or \
+                         metrics.get("summaryDTO", {}).get("averageHR", None)
+                if avg_hr is not None:
+                    activity.avg_heart_rate = int(float(avg_hr))
+                
+                # Extract power and calories
+                avg_power = metrics.get("summaryDTO", {}).get("avgPower")
+                if avg_power is not None:
+                    activity.avg_power = float(avg_power)
+                
+                calories = metrics.get("summaryDTO", {}).get("calories")
+                if calories is not None:
+                    activity.calories = int(float(calories))
+            
+            # Update sync timestamp
+            activity.last_sync = datetime.now().isoformat()
 
         session.commit()
     except SQLAlchemyError as e:

garminsync/migrate_activities.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 """
-Migration script to populate new activity fields from Garmin API
+Migration script to populate new activity fields from FIT files or Garmin API
 """
 
 import os
@@ -16,11 +16,22 @@ sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
 from garminsync.database import Activity, get_session, init_db
 from garminsync.garmin import GarminClient
+from garminsync.activity_parser import get_activity_metrics
 
 
 def add_columns_to_database():
     """Add new columns to the activities table if they don't exist"""
-    print("Adding new columns to database...")
+
+# Add the parent directory to the path to import garminsync modules
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from garminsync.database import Activity, get_session, init_db
+from garminsync.garmin import GarminClient
+
+
+def add_columns_to_database():
+    """Add new columns to the activities table if they don't exist"""
+    print("Adding new columns to database...", flush=True)
 
     # Get database engine
     db_path = os.path.join(os.getenv("DATA_DIR", "data"), "garmin.db")
@@ -49,7 +60,7 @@ def add_columns_to_database():
         with engine.connect() as conn:
             for column_name in new_columns:
                 if column_name not in existing_columns:
-                    print(f"Adding column {column_name}...")
+                    print(f"Adding column {column_name}...", flush=True)
                     if column_name in ["distance", "avg_power"]:
                         conn.execute(
                             text(
@@ -69,21 +80,22 @@ def add_columns_to_database():
                             )
                         )
                     conn.commit()
-                    print(f"Column {column_name} added successfully")
+                    print(f"Column {column_name} added successfully", flush=True)
                 else:
-                    print(f"Column {column_name} already exists")
+                    print(f"Column {column_name} already exists", flush=True)
 
-        print("Database schema updated successfully")
+        print("Database schema updated successfully", flush=True)
         return True
 
     except Exception as e:
-        print(f"Failed to update database schema: {e}")
+        print(f"Failed to update database schema: {e}", flush=True)
         return False
 
 
+
 def migrate_activities():
-    """Migrate activities to populate new fields from Garmin API"""
-    print("Starting activity migration...")
+    """Migrate activities to populate new fields from FIT files or Garmin API"""
+    print("Starting activity migration...", flush=True)
 
     # First, add columns to database
     if not add_columns_to_database():
@@ -92,9 +104,9 @@ def migrate_activities():
     # Initialize Garmin client
     try:
         client = GarminClient()
-        print("Garmin client initialized successfully")
+        print("Garmin client initialized successfully", flush=True)
     except Exception as e:
-        print(f"Failed to initialize Garmin client: {e}")
+        print(f"Failed to initialize Garmin client: {e}", flush=True)
         # Continue with migration but without Garmin data
         client = None
 
@@ -106,12 +118,12 @@ def migrate_activities():
         activities = (
             session.query(Activity).filter(Activity.activity_type.is_(None)).all()
         )
-        print(f"Found {len(activities)} activities to migrate")
+        print(f"Found {len(activities)} activities to migrate", flush=True)
 
         # If no activities found, try to get all activities (in case activity_type column was just added)
         if len(activities) == 0:
             activities = session.query(Activity).all()
-            print(f"Found {len(activities)} total activities")
+            print(f"Found {len(activities)} total activities", flush=True)
 
         updated_count = 0
         error_count = 0
@@ -119,13 +131,16 @@ def migrate_activities():
         for i, activity in enumerate(activities):
             try:
                 print(
-                    f"Processing activity {i+1}/{len(activities)} (ID: {activity.activity_id})"
+                    f"Processing activity {i+1}/{len(activities)} (ID: {activity.activity_id})", 
+                    flush=True
                 )
 
-                # Fetch detailed activity data from Garmin (if client is available)
-                activity_details = None
-                if client:
-                    activity_details = client.get_activity_details(activity.activity_id)
+                # Use shared parser to get activity metrics
+                activity_details = get_activity_metrics(activity, client)
+                if activity_details is not None:
+                    print(f"  Successfully parsed metrics for activity {activity.activity_id}", flush=True)
+                else:
+                    print(f"  Could not retrieve metrics for activity {activity.activity_id}", flush=True)
 
                 # Update activity fields if we have details
                 if activity_details:
@@ -170,19 +185,19 @@ def migrate_activities():
 
                 # Print progress every 10 activities
                 if (i + 1) % 10 == 0:
-                    print(f"  Progress: {i+1}/{len(activities)} activities processed")
+                    print(f"  Progress: {i+1}/{len(activities)} activities processed", flush=True)
 
             except Exception as e:
-                print(f"  Error processing activity {activity.activity_id}: {e}")
+                print(f"  Error processing activity {activity.activity_id}: {e}", flush=True)
                 session.rollback()
                 error_count += 1
                 continue
 
-        print(f"Migration completed. Updated: {updated_count}, Errors: {error_count}")
+        print(f"Migration completed. Updated: {updated_count}, Errors: {error_count}", flush=True)
         return True  # Allow partial success
 
     except Exception as e:
-        print(f"Migration failed: {e}")
+        print(f"Migration failed: {e}", flush=True)
         return False
     finally:
         session.close()

justfile

@@ -9,11 +9,10 @@ dev:
     just build
     docker run -it --rm --env-file .env -v $(pwd)/garminsync:/app/garminsync -v $(pwd)/data:/app/data -p 8888:8888 --name garminsync-dev garminsync uvicorn garminsync.web.app:app --reload --host 0.0.0.0 --port 8080
 
-# Run database migrations (container-based)
+# Run database migrations with enhanced logging (container-based)
 migrate:
     just build
-    docker run --rm --env-file .env -v $(pwd)/data:/app/data --entrypoint "alembic" garminsync upgrade head
-
+    docker run --rm --env-file .env -v $(pwd)/data:/app/data --entrypoint "python" garminsync -m garminsync.cli migrate
 # Run validation tests (container-based)
 test:
     just build
@@ -41,7 +40,7 @@ format:
 # Start production server
 run_server:
     just build
-    docker run -d --rm --env-file .env -v $(pwd)/data:/app/data -p 8888:8888 --name garminsync garminsync daemon --start
+    docker run -d --rm --env-file .env -e RUN_MIGRATIONS=1 -v $(pwd)/data:/app/data -p 8888:8888 --name garminsync garminsync daemon --start
 
 # Stop production server
 stop_server:
@@ -50,7 +49,7 @@ stop_server:
 # Run server in live mode for debugging
 run_server_live:
     just build
-    docker run -it --rm --env-file .env -v $(pwd)/data:/app/data -p 8888:8888 --name garminsync garminsync daemon --start
+    docker run -it --rm --env-file .env -e RUN_MIGRATIONS=1 -v $(pwd)/data:/app/data -p 8888:8888 --name garminsync garminsync daemon --start
 
 # Clean up any existing container
 cleanup:

requirements.txt

@@ -17,3 +17,4 @@ tqdm==4.66.1
 sqlalchemy==2.0.30
 pylint==3.1.0
 pygments==2.18.0
+fitdecode

tests/test_sync.py

@@ -1,102 +1,110 @@
 import pytest
 import sys
-from unittest.mock import Mock, patch
+from unittest.mock import Mock, patch, MagicMock
 
 # Add the project root to the Python path
 sys.path.insert(0, '/app')
 
-from garminsync.database import sync_database
-from garminsync.garmin import GarminClient
-
+from garminsync.database import sync_database, Activity, get_activity_metrics
 
 def test_sync_database_with_valid_activities():
     """Test sync_database with valid API response"""
-    mock_client = Mock(spec=GarminClient)
+    mock_client = Mock()
     mock_client.get_activities.return_value = [
         {"activityId": 12345, "startTimeLocal": "2023-01-01T10:00:00"},
         {"activityId": 67890, "startTimeLocal": "2023-01-02T11:00:00"}
     ]
     
-    with patch('garminsync.database.get_session') as mock_session:
-        mock_session.return_value.query.return_value.filter_by.return_value.first.return_value = None
+    mock_session = MagicMock()
+    mock_session.query.return_value.filter_by.return_value.first.return_value = None
+    
+    with patch('garminsync.database.get_session', return_value=mock_session), \
+         patch('garminsync.database.get_activity_metrics', return_value={
+             "activityType": {"typeKey": "running"},
+             "summaryDTO": {
+                 "duration": 3600,
+                 "distance": 10.0,
+                 "maxHR": 180,
+                 "calories": 400
+             }
+         }):
         
         sync_database(mock_client)
         
-        # Verify get_activities was called
-        mock_client.get_activities.assert_called_once_with(0, 1000)
-        
-        # Verify database operations
-        mock_session.return_value.add.assert_called()
-        mock_session.return_value.commit.assert_called()
-
+        # Verify activities processed
+        assert mock_session.add.call_count == 2
+        assert mock_session.commit.called
 
 def test_sync_database_with_none_activities():
     """Test sync_database with None response from API"""
-    mock_client = Mock(spec=GarminClient)
+    mock_client = Mock()
     mock_client.get_activities.return_value = None
     
-    with patch('garminsync.database.get_session') as mock_session:
+    mock_session = MagicMock()
+    
+    with patch('garminsync.database.get_session', return_value=mock_session):
         sync_database(mock_client)
-        
-        # Verify get_activities was called
-        mock_client.get_activities.assert_called_once_with(0, 1000)
-        
-        # Verify no database operations
-        mock_session.return_value.add.assert_not_called()
-        mock_session.return_value.commit.assert_not_called()
-
+        mock_session.add.assert_not_called()
 
 def test_sync_database_with_missing_fields():
     """Test sync_database with activities missing required fields"""
-    mock_client = Mock(spec=GarminClient)
+    mock_client = Mock()
     mock_client.get_activities.return_value = [
-        {"activityId": 12345},  # Missing startTimeLocal
-        {"startTimeLocal": "2023-01-02T11:00:00"},  # Missing activityId
-        {"activityId": 67890, "startTimeLocal": "2023-01-03T12:00:00"}  # Valid
+        {"activityId": 12345},
+        {"startTimeLocal": "2023-01-02T11:00:00"},
+        {"activityId": 67890, "startTimeLocal": "2023-01-03T12:00:00"}
     ]
     
-    with patch('garminsync.database.get_session') as mock_session:
-        mock_session.return_value.query.return_value.filter_by.return_value.first.return_value = None
-        
+    # Create a mock that returns None for existing activity
+    mock_session = MagicMock()
+    mock_session.query.return_value.filter_by.return_value.first.return_value = None
+    
+    with patch('garminsync.database.get_session', return_value=mock_session), \
+         patch('garminsync.database.get_activity_metrics', return_value={
+             "summaryDTO": {"duration": 3600.0}
+         }):
         sync_database(mock_client)
-        
-        # Verify only one activity was added (the valid one)
-        assert mock_session.return_value.add.call_count == 1
-        mock_session.return_value.commit.assert_called()
-
+        # Only valid activity should be added
+        assert mock_session.add.call_count == 1
+        added_activity = mock_session.add.call_args[0][0]
+        assert added_activity.activity_id == 67890
 
 def test_sync_database_with_existing_activities():
     """Test sync_database doesn't duplicate existing activities"""
-    mock_client = Mock(spec=GarminClient)
+    mock_client = Mock()
     mock_client.get_activities.return_value = [
         {"activityId": 12345, "startTimeLocal": "2023-01-01T10:00:00"}
     ]
     
-    with patch('garminsync.database.get_session') as mock_session:
-        # Mock existing activity
-        mock_session.return_value.query.return_value.filter_by.return_value.first.return_value = Mock()
-        
+    mock_session = MagicMock()
+    mock_session.query.return_value.filter_by.return_value.first.return_value = Mock()
+    
+    with patch('garminsync.database.get_session', return_value=mock_session), \
+         patch('garminsync.database.get_activity_metrics', return_value={
+             "summaryDTO": {"duration": 3600.0}
+         }):
         sync_database(mock_client)
-        
-        # Verify no new activities were added
-        mock_session.return_value.add.assert_not_called()
-        mock_session.return_value.commit.assert_called()
-
+        mock_session.add.assert_not_called()
 
 def test_sync_database_with_invalid_activity_data():
     """Test sync_database with invalid activity data types"""
-    mock_client = Mock(spec=GarminClient)
+    mock_client = Mock()
     mock_client.get_activities.return_value = [
-        "invalid activity data",  # Not a dict
-        None,  # None value
-        {"activityId": 12345, "startTimeLocal": "2023-01-01T10:00:00"}  # Valid
+        "invalid data",
+        None,
+        {"activityId": 12345, "startTimeLocal": "2023-01-01T10:00:00"}
     ]
     
-    with patch('garminsync.database.get_session') as mock_session:
-        mock_session.return_value.query.return_value.filter_by.return_value.first.return_value = None
-        
+    # Create a mock that returns None for existing activity
+    mock_session = MagicMock()
+    mock_session.query.return_value.filter_by.return_value.first.return_value = None
+    
+    with patch('garminsync.database.get_session', return_value=mock_session), \
+         patch('garminsync.database.get_activity_metrics', return_value={
+             "summaryDTO": {"duration": 3600.0}
+         }):
         sync_database(mock_client)
-        
-        # Verify only one activity was added (the valid one)
-        assert mock_session.return_value.add.call_count == 1
-        mock_session.return_value.commit.assert_called()
+        # Only valid activity should be added
+        assert mock_session.add.call_count == 1
+        added_activity = mock_session.add.call_args[0][0]
+        assert added_activity.activity_id == 12345

workflows.md

@@ -0,0 +1,100 @@
+# GarminSync Workflows
+
+## Migration Workflow
+
+### Purpose
+Add new columns to database and populate with activity metrics
+
+### Trigger
+`python cli.py migrate`
+
+### Steps
+1. Add required columns to activities table:
+   - activity_type (TEXT)
+   - duration (INTEGER)
+   - distance (REAL)
+   - max_heart_rate (INTEGER)
+   - avg_power (REAL)
+   - calories (INTEGER)
+2. For each activity:
+   - Parse metrics from local FIT/XML files
+   - Fetch from Garmin API if local files missing
+   - Update database fields
+3. Commit changes
+4. Report migration status
+
+### Error Handling
+- Logs errors per activity
+- Marks unprocessable activities as "Unknown"
+- Continues processing other activities on error
+
+## Sync Workflow
+
+### Purpose
+Keep local database synchronized with Garmin Connect
+
+### Triggers
+- CLI commands (`list`, `download`)
+- Scheduled daemon (every 6 hours by default)
+- Web UI requests
+
+### Core Components
+- `sync_database()`: Syncs activity metadata
+- `download()`: Fetches missing FIT files
+- Daemon: Background scheduler and web UI
+
+### Process Flow
+1. Authenticate with Garmin API
+2. Fetch latest activities
+3. For each activity:
+   - Parse metrics from FIT/XML files
+   - Fetch from Garmin API if local files missing
+   - Update database fields
+4. Download missing activity files
+5. Update sync timestamps
+6. Log operations
+
+### Database Schema
+```mermaid
+erDiagram
+    activities {
+        integer activity_id PK
+        string start_time
+        string activity_type
+        integer duration
+        float distance
+        integer max_heart_rate
+        integer avg_heart_rate
+        float avg_power
+        integer calories
+        string filename
+        boolean downloaded
+        string created_at
+        string last_sync
+    }
+    
+    daemon_config {
+        integer id PK
+        boolean enabled
+        string schedule_cron
+        string last_run
+        string next_run
+        string status
+    }
+    
+    sync_logs {
+        integer id PK
+        string timestamp
+        string operation
+        string status
+        string message
+        integer activities_processed
+        integer activities_downloaded
+    }
+```
+
+### Key Notes
+- Data directory: `data/` (configurable via DATA_DIR)
+- Web UI port: 8080 (default)
+- Downloaded files: `activity_{id}_{timestamp}.fit`
+- Metrics include: heart rate, power, calories, distance