3.05 – AI Operations (AIOps) | AI Course

📈 3.05 – AI Operations (AIOps)

📑 On this page

📊 Monitoring & Observability

📌 What to Monitor in Production AI:
Structured Logging with OpenTelemetry: 
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
import time

# Setup tracing
provider = TracerProvider()
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4317"))
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

tracer = trace.get_tracer(__name__)

def llm_call_with_tracing(prompt):
    start = time.time()
    with tracer.start_as_current_span("llm_completion") as span:
        response = call_llm(prompt)
        latency_ms = (time.time() - start) * 1000
        
        span.set_attribute("prompt_length", len(prompt))
        span.set_attribute("response_length", len(response))
        span.set_attribute("latency_ms", latency_ms)
        span.set_attribute("tokens_used", response.usage.total_tokens)
        
        return response
📌 Recommended AI Observability Tools:
💡 Health Check Dashboard Metrics:

💰 Cost Management

📌 Cost Optimization Strategies:
Daily Cost Budget Enforcer: 
import redis
import time

class BudgetManager:
    def __init__(self, daily_budget_usd, redis_client):
        self.daily_budget = daily_budget_usd
        self.redis = redis_client
        self.today = time.strftime("%Y-%m-%d")
    
    def check_and_record(self, cost_usd):
        key = f"daily_spend:{self.today}"
        current = float(self.redis.get(key) or 0)
        
        if current + cost_usd > self.daily_budget:
            return False  # Block request
        else:
            self.redis.incrbyfloat(key, cost_usd)
            return True  # Allow request
📌 Cost Dashboard Example:
MetricValueAlert Threshold
Daily spend $42.50 $100 (🟢 OK)
Tokens per request (avg) 1,250 >1,500 (🟡 Warn)
Cost per request (avg) $0.0085 >$0.01 (🟢 OK)
Most expensive user User #1234: $12.20 Alert if >$20

📦 Model & Prompt Versioning

📌 Treat Prompts Like Code:
  • Store prompts in version control (Git)
  • Use semantic versioning (v1.0, v1.1)
  • Tag each deployment with the prompt version
  • Maintain change log for all prompt updates
Prompt Registry with Versioning: 
import json
from datetime import datetime

class PromptRegistry:
    def __init__(self, storage_dir="prompts"):
        self.storage_dir = storage_dir
    
    def register(self, name, prompt_template, description):
        version = self._next_version(name)
        entry = {
            "name": name,
            "version": version,
            "prompt": prompt_template,
            "description": description,
            "created_at": datetime.utcnow().isoformat(),
            "deployed": False
        }
        with open(f"{self.storage_dir}/{name}_{version}.json", "w") as f:
            json.dump(entry, f)
        return version
    
    def deploy(self, name, version):
        # Mark as deployed in production
        pass
    
    def rollback(self, name):
        # Revert to previous version
        pass

🧪 A/B Testing & Canary Deployments

📌 A/B Test Configuration: 
class ABTest:
    def __init__(self, config):
        # config = {"variant_a": 50%, "variant_b": 50%}
        self.config = config
        self.results = {"variant_a": [], "variant_b": []}
    
    def get_variant(self, user_id):
        # Deterministic assignment based on user_id
        hash_val = hash(user_id) % 100
        if hash_val < self.config["variant_a"]:
            return "variant_a"
        return "variant_b"
    
    def record_result(self, variant, score):
        self.results[variant].append(score)
    
    def get_winner(self):
        # Statistical significance check
        from scipy import stats
        a_scores = self.results["variant_a"]
        b_scores = self.results["variant_b"]
        _, p_value = stats.ttest_ind(a_scores, b_scores)
        if p_value < 0.05:
            return "variant_b" if np.mean(b_scores) > np.mean(a_scores) else "variant_a"
        return None  # inconclusive
📌 Canary Deployment Strategy:
  • 1% users → new model/prompt
  • Monitor error rate, latency, quality for 24 hours
  • If metrics acceptable, increase to 10%
  • Monitor another 24 hours
  • Gradual rollout to 25%, 50%, 100%
  • If issues detected, instant rollback

🚨 Incident Response

📌 AI-Specific Incident Scenarios:
  • Degraded quality: Model drift, prompt regression
  • Increased cost: Unexpected token usage, infinite loops
  • API outages: Downstream LLM provider issues
  • Safety violations: Harmful outputs detected
  • Rate limiting: Hitting API limits
Incident Response Playbook Template: 
INCIDENT_RESPONSE = {
    "detection": [
        "Automated alert (latency > 10s for 5 min)",
        "User report via feedback system"
    ],
    "triage": [
        "Check if issue affects all users or subset",
        "Verify API provider status page",
        "Compare metrics to baseline"
    ],
    "mitigation": [
        "Rollback to previous prompt version",
        "Switch to fallback model",
        "Enable rate limiting to protect system"
    ],
    "resolution": [
        "RCA within 24 hours",
        "Deploy fix (prompt update, model update)",
        "Post-mortem documentation"
    ]
}
💡 Runbooks Checklist:
  • ✅ How to roll back a prompt (git revert + redeploy)
  • ✅ How to switch to a backup model (e.g., GPT-4 → GPT-3.5)
  • ✅ Who to notify on-call (escalation list)
  • ✅ SLA commitments (e.g., 99.9% uptime, 5s p95 latency)
  • ✅ Post-incident review template
⚠️ Production Readiness Checklist:
  • ✅ Monitoring and alerting configured
  • ✅ Cost budgets and alerts set
  • ✅ Prompts versioned and rollback capable
  • ✅ A/B testing framework for continuous improvement
  • ✅ Incident runbooks documented and tested
  • ✅ Load testing completed for expected traffic

✍️ Exercises

Exercise 3.05.1 – Design metrics dashboard

List 5 metrics you would track for an AI-powered customer support chatbot. For each, define the healthy range and alert threshold.

1. Response time (p95): <5s healthy, >10s alert
2. Resolution rate: >40% healthy, <20% alert
3. Cost per conversation: <$0.10 healthy, >$0.25 alert
4. User satisfaction: >85% healthy, <70% alert
5. API error rate: <1% healthy, >5% alert
Exercise 3.05.2 – A/B test design

You have two prompts for summarization. Design an A/B test to determine which is better. Include: metrics, sample size, duration.

Design:
- Metrics: Summary relevance (1-10 from user rating), output length, latency
- Sample size: 1000 users per variant (detects 10% difference at 95% confidence)
- Duration: 3-5 days (depending on traffic)
- Randomization: User ID hash
- Winner criteria: Statistical significance (p < 0.05) + business relevance
Exercise 3.05.3 – Incident simulation

You see a sudden 300% increase in token usage. Walk through your incident response steps.

Response:
1. Alert triggers (automated) → page on-call
2. Triage: Check if from single user or global. Look at logs.
3. Mitigation: Deploy previous prompt version. Add rate limiting for offending user.
4. Resolution: Root cause = new prompt causing infinite loops. Roll back, fix prompt, redeploy.
5. Post-incident: Add prompt validation to CI/CD, set cost alert at 200% baseline.
📌 Key Takeaways
  • Monitoring = know when things are wrong (latency, cost, errors)
  • Cost management is critical – LLMs are cheap per token but add up fast
  • Version everything: prompts, models, configurations
  • A/B testing enables continuous improvement with low risk
  • Have runbooks ready before incidents happen
  • Post-incident reviews prevent repeat issues
🎉 Level 3 Complete!

You've finished Level 3 of the AI course. You now have professional-level knowledge to:

  • ✅ Develop enterprise AI strategy and ROI models
  • ✅ Implement security, compliance, and guardrails
  • ✅ Design multi-agent systems for complex tasks
  • ✅ Fine-tune models for specialized domains
  • ✅ Operate AI systems in production with monitoring and cost controls

Certificate of completion available! 🏆

© AI Course – 3.05: AI Operations (AIOps) – Level 3 Complete!