Skip to main content

Training Monitoring

Track training progress, monitor metrics, analyze results, and manage checkpoints in real-time.

Training Logs​

View training logs in real-time:

Log Information:

  • Training Progress: Real-time training progress
  • Error Messages: Error and warning messages
  • System Logs: System and infrastructure logs
  • Job Status: Training job status updates

Log Features:

  • Real-Time Updates: Live log streaming
  • Log Filtering: Filter logs by level or type
  • Log Search: Search through logs
  • Log Export: Export logs for analysis

Training Metrics​

Monitor training metrics:

Metrics Tracked:

  • Loss: Training and validation loss
  • Accuracy: Model accuracy (for classification)
  • Other Metrics: Task-specific metrics
  • Learning Rate: Current learning rate
  • Epoch Progress: Current epoch and progress

Metrics Visualization:

  • Line Charts: Metrics over time
  • Comparison: Training vs validation metrics
  • Trends: Identify trends and patterns
  • Alerts: Alerts for metric thresholds

Training Summary​

View training summary:

Summary Information:

  • Training Status: Current training status
  • Duration: Training duration
  • Progress: Overall progress percentage
  • Checkpoints: Number of checkpoints created
  • Best Metrics: Best validation metrics achieved

Summary Metrics:

  • Final Loss: Final training and validation loss
  • Best Epoch: Epoch with best validation metrics
  • Training Time: Total training time
  • Resource Usage: GPU and memory usage

Feature Analysis During Training​

Analyze features during training:

Feature Analysis:

  • Feature Importance: Feature importance scores
  • Feature Contributions: Feature contributions to predictions
  • Feature Interactions: Feature interaction analysis
  • Feature Trends: Feature trends over training

Analysis Tools:

  • Feature Importance Charts: Visualize feature importance
  • Contribution Analysis: Analyze feature contributions
  • Interaction Plots: Visualize feature interactions
  • Trend Analysis: Track feature trends

Checkpoint Tracking​

Track and manage checkpoints:

Checkpoint Information:

  • Checkpoint List: List of all checkpoints
  • Checkpoint Metrics: Metrics for each checkpoint
  • Best Checkpoint: Best performing checkpoint
  • Checkpoint Details: Detailed checkpoint information

Checkpoint Operations:

  • View Checkpoints: View all checkpoints
  • Compare Checkpoints: Compare checkpoint performance
  • Select Best: Identify best checkpoint
  • Download Checkpoints: Download checkpoint files

Next Steps​