Observing Changes in the Codebase#
#TODO:
Add an example for a continuos monitoring of a Huggingface artefact
Create an external demo repo and include all these steps
Designing and configuring observer agents to monitor each possible pipeline action requires a comprehensive approach to ensure that every aspect of your CI/CD pipeline is observed for changes. Observer agents can help in detecting code changes, data modifications, feature engineering updates, model training variations, and deployment adjustments. Here’s a detailed design and configuration plan:
Observer Agents of Different Stages#
0. Tools#
GitHub/GitLab Configuration: Set up workflows to trigger on code changes.
Airflow: Define DAGs with sensors for data and feature changes.
MLflow: Log model training and validation metrics.
Kubernetes: Monitor deployment and infrastructure.
Prometheus/Grafana: Monitor post-deployment metrics and set up alerts.
1. Codebase Monitoring Agent#
Purpose: To monitor changes in the codebase, such as updates to scripts, configuration files, and dependency changes.
Tools: Git hooks, GitHub Actions, GitLab CI/CD.
Configuration:
Git Hooks: Set up pre-commit and post-commit hooks to log changes and run initial checks.
GitHub Actions / GitLab CI: Configure workflows to trigger on pull requests, commits, and merges.
Example GitHub Action:
name: Codebase Monitoring
on: [push, pull_request]
jobs:
monitor:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v2
- name: Run Code Linter
run: |
flake8 .
- name: Dependency Check
run: |
pip check
- name: Notify
run: |
echo "Codebase has been updated and checked."
2. Data Pipeline Monitoring Agent#
Purpose: To monitor changes in data sources, data processing scripts, and data quality.
Tools: Apache Airflow, Great Expectations, Data Version Control (DVC).
Configuration:
Airflow DAG: Define DAGs to manage data workflows and set up sensors to detect changes.
Great Expectations: Integrate data validation checks in the pipeline.
Example Airflow Sensor:
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from airflow.sensors.filesystem import FileSensor
from datetime import datetime
def data_validation():
import great_expectations as ge
# Your Great Expectations data validation logic here
default_args = {
'start_date': datetime(2023, 1, 1),
}
dag = DAG('data_pipeline_monitor', default_args=default_args, schedule_interval='@daily')
t1 = FileSensor(
task_id='check_for_new_data',
filepath='/path/to/data',
poke_interval=10,
timeout=600,
dag=dag,
)
t2 = PythonOperator(
task_id='validate_data',
python_callable=data_validation,
dag=dag,
)
t1 >> t2
3. Feature Engineering Monitoring Agent#
Purpose: To observe changes in feature engineering scripts and the generated features.
Tools: Git hooks, CI/CD tools, Data Version Control (DVC).
Configuration:
Git Hooks: Monitor changes in the
src/featuresdirectory.CI/CD Integration: Include steps in CI/CD pipelines to validate feature scripts.
Example CI Step:
- name: Check Feature Engineering
run: |
python src/features/feature_checks.py
4. Model Training and Validation Monitoring Agent#
Purpose: To monitor changes in model training scripts, hyperparameters, and validation results.
Tools: MLflow, GitHub Actions, Jenkins.
Configuration:
MLflow Tracking: Use MLflow to log parameters, metrics, and artifacts.
CI/CD Integration: Automate training and validation steps in the pipeline.
Example MLflow Integration:
import mlflow
with mlflow.start_run():
mlflow.log_param("param1", value)
mlflow.log_metric("metric1", value)
mlflow.log_artifact("model.pkl")
5. Deployment Monitoring Agent#
Purpose: To monitor deployment scripts and infrastructure changes.
Tools: Kubernetes, Docker, Prometheus, Grafana.
Configuration:
Kubernetes Manifests: Monitor changes in deployment manifests.
CI/CD Integration: Automate deployment checks and validations.
Example Kubernetes Monitoring:
apiVersion: v1
kind: ConfigMap
metadata:
name: deployment-monitor
data:
monitor.sh: |
#!/bin/bash
kubectl get pods
kubectl get services
6. Post-Deployment Monitoring Agent#
Purpose: To observe the model’s performance and fairness in production.
Tools: Evidently, WhyLabs, Prometheus, Grafana.
Configuration:
Monitoring Dashboards: Set up dashboards to visualize key metrics.
Alerts: Configure alerts for any deviations in performance or fairness metrics.
Example Evidently Integration:
import evidently
from evidently.dashboard import Dashboard
from evidently.tabs import DataDriftTab, CatTargetDriftTab
dashboard = Dashboard(tabs=[DataDriftTab(), CatTargetDriftTab()])
dashboard.calculate(reference_data, production_data)
dashboard.save("dashboard.html")
Summary#
By setting up these observer agents and integrating them into your CI/CD pipeline, you can proactively monitor changes at every stage of your ML pipeline. This approach ensures that you maintain control over your pipeline’s integrity and can quickly identify and address any issues, including bias and performance deviations.
And the UML Sequence Diagram for this workflow is: