跳转至

第五天:应用实践与部署运维

学习目标

  • 掌握多智能体系统的完整开发流程
  • 学会企业级部署与运维最佳实践
  • 具备独立构建多智能体系统的能力
  • 了解生产环境的监控与优化策略
  • 掌握容器化和云原生部署技术

参考项目

本课程将带领学员完成一个完整的企业级多智能体系统项目,从需求分析到部署运维的全流程实践。

💡 实际项目参考:本课程基于项目中的完整实现进行教学:

  • src/examples/customer_service_system.py - 完整的企业级客服系统实现
  • main.py - 系统启动和部署脚本
  • docker-compose.yml - 容器化部署配置
  • docker-compose.yml - Kubernetes部署清单
  • src/monitoring/ - 监控和告警配置

实战特色:从实际企业需求出发,涵盖完整的开发、测试、部署、监控、运维流程,提供生产级的解决方案。


1. 综合项目实战

1.1 项目需求分析

1.1.1 基于实际项目的智能客服系统

**代码引用**: 完整的智能客服系统实现请参考 `multi_agent_system/src/examples/customer_service_system.py`

基于实际项目的智能客服系统需求分析:

**核心数据结构**:
- `CustomerServicePriority`: 客服优先级管理(低、中、高、紧急)
- `TicketStatus`: 工单状态跟踪(开放、进行中、待处理、已解决、已关闭)
- `CustomerSentiment`: 客户情绪分析(非常负面到非常正面)
- `CustomerProfile`: 客户档案管理(VIP等级、历史记录、满意度评分)
- `SupportTicket`: 支持工单完整生命周期管理

**功能需求**:
- 智能对话:多轮对话,意图识别准确率 > 95%
- 知识检索:向量数据库 + 语义搜索,响应时间 < 500ms
- 工单处理:工作流引擎 + 状态机,处理效率提升 50%
- 情感分析:情感识别准确率 > 90%,智能转人工
- 多渠道接入:支持 5+ 渠道(网页、微信、电话等)
- VIP客户识别:优先级队列,响应时间 < 30秒

**非功能需求**:
- 性能:响应时间 < 2秒,并发用户 > 1000,吞吐量 > 10000 QPS
- 可用性:99.9% 系统可用性,7x24小时服务,故障恢复 < 5分钟
- 安全性:数据加密、RBAC权限管理、完整审计日志
- 可扩展性:自动扩容、插件化架构、多租户支持

# 实际项目架构实现
class ProductionSystemArchitecture:
    """生产级系统架构(基于实际项目)"""

    def __init__(self):
        self.core_agents = {
            "CustomerServiceAgent": {
                "file": "customer_service_system.py",
                "description": "主要客服智能体",
                "capabilities": [
                    "客户咨询处理",
                    "工单管理",
                    "情感分析",
                    "知识检索",
                    "VIP客户识别"
                ]
            },
            "BaseAgent": {
                "file": "base_agent.py", 
                "description": "BDI智能体基础架构",
                "capabilities": [
                    "信念管理",
                    "愿望规划",
                    "意图执行",
                    "状态监控"
                ]
            }
        }

        self.infrastructure_components = {
            "消息总线": {
                "file": "message_bus.py",
                "technology": "异步消息队列",
                "features": ["发布订阅", "消息持久化", "负载均衡"]
            },
            "工作流引擎": {
                "file": "langgraph_workflow.py",
                "technology": "LangGraph",
                "features": ["状态管理", "条件分支", "错误恢复"]
            },
            "监控系统": {
                "file": "langsmith_integration.py",
                "technology": "LangSmith",
                "features": ["链路追踪", "性能监控", "告警系统"]
            },
            "数据存储": {
                "components": ["PostgreSQL", "Redis", "Elasticsearch"],
                "features": ["关系数据", "缓存", "全文搜索"]
            }
        }

        self.deployment_architecture = {
            "容器化": {
                "technology": "Docker + Kubernetes",
                "benefits": ["环境一致性", "快速部署", "弹性扩容"]
            },
            "微服务": {
                "pattern": "每个智能体独立服务",
                "benefits": ["独立扩展", "故障隔离", "技术多样性"]
            },
            "云原生": {
                "features": ["自动扩容", "服务发现", "配置管理"],
                "platforms": ["Kubernetes", "Docker Swarm", "云平台"]
            }
        }

# 实际部署配置示例
class ProductionDeploymentConfig:
    """生产环境部署配置"""

    @staticmethod
    def get_docker_compose_config():
        """Docker Compose 配置"""
        return """
version: '3.8'

services:
  # 主应用服务
  multi-agent-system:
    build: .
    ports:
      - "8000:8000"
    environment:
      - ENVIRONMENT=production
      - DATABASE_URL=postgresql://user:pass@postgres:5432/multiagent
      - REDIS_URL=redis://redis:6379
      - LANGSMITH_API_KEY=${LANGSMITH_API_KEY}
    depends_on:
      - postgres
      - redis
      - elasticsearch
    volumes:
      - ./logs:/app/logs
      - ./data:/app/data
    restart: unless-stopped

  # 数据库服务
  postgres:
    image: postgres:15
    environment:
      - POSTGRES_DB=multiagent
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=pass
    volumes:
      - postgres_data:/var/lib/postgresql/data
    ports:
      - "5432:5432"
    restart: unless-stopped

  # 缓存服务
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data
    restart: unless-stopped

  # 搜索引擎
  elasticsearch:
    image: elasticsearch:8.8.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
    ports:
      - "9200:9200"
    volumes:
      - es_data:/usr/share/elasticsearch/data
    restart: unless-stopped

  # 监控服务
  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    restart: unless-stopped

  # 可视化面板
  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana_data:/var/lib/grafana
      - ./monitoring/grafana:/etc/grafana/provisioning
    restart: unless-stopped

volumes:
  postgres_data:
  redis_data:
  es_data:
  prometheus_data:
  grafana_data:
"""

    @staticmethod
    def get_kubernetes_config():
        """Kubernetes 部署配置"""
        return """
apiVersion: apps/v1
kind: Deployment
metadata:
  name: multi-agent-system
  labels:
    app: multi-agent-system
spec:
  replicas: 3
  selector:
    matchLabels:
      app: multi-agent-system
  template:
    metadata:
      labels:
        app: multi-agent-system
    spec:
      containers:
      - name: multi-agent-system
        image: multi-agent-system:latest
        ports:
        - containerPort: 8000
        env:
        - name: ENVIRONMENT
          value: "production"
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: db-secret
              key: url
        - name: REDIS_URL
          value: "redis://redis-service:6379"
        - name: LANGSMITH_API_KEY
          valueFrom:
            secretKeyRef:
              name: langsmith-secret
              key: api-key
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "1Gi"
            cpu: "500m"
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 8000
          initialDelaySeconds: 5
          periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: multi-agent-system-service
spec:
  selector:
    app: multi-agent-system
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8000
  type: LoadBalancer
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: multi-agent-system-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: multi-agent-system
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
"""

1.2 生产环境部署实践

1.2.1 容器化部署

# 基于实际项目的 Dockerfile
class ProductionDockerfile:
    """生产级 Dockerfile 配置"""

    @staticmethod
    def get_dockerfile():
        return """
# 多阶段构建,优化镜像大小
FROM python:3.11-slim as builder

# 设置工作目录
WORKDIR /app

# 安装系统依赖
RUN apt-get update && apt-get install -y \\
    gcc \\
    g++ \\
    && rm -rf /var/lib/apt/lists/*

# 复制依赖文件
COPY requirements.txt .

# 安装Python依赖
RUN pip install --no-cache-dir --user -r requirements.txt

# 生产阶段
FROM python:3.11-slim

# 创建非root用户
RUN useradd --create-home --shell /bin/bash app

# 设置工作目录
WORKDIR /app

# 从builder阶段复制依赖
COPY --from=builder /root/.local /home/app/.local

# 复制应用代码
COPY . .

# 设置权限
RUN chown -R app:app /app

# 切换到非root用户
USER app

# 设置环境变量
ENV PATH=/home/app/.local/bin:$PATH
ENV PYTHONPATH=/app

# 健康检查
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \\
    CMD python -c "import requests; requests.get('http://localhost:8000/health')"

# 暴露端口
EXPOSE 8000

# 启动命令
CMD ["python", "main.py"]
"""

# 生产环境启动脚本
class ProductionStartupScript:
    """生产环境启动脚本"""

    @staticmethod
    def get_startup_script():
        return """
#!/bin/bash

# 基于项目 multi_agent_system/main.py 的启动流程

set -e

echo "Starting Multi-Agent System in Production Mode..."

# 环境检查
echo "Checking environment..."
python -c "
import sys
import os
from src.agents.base_agent import BaseAgent
from src.examples.customer_service_system import CustomerServiceAgent
from src.communication.message_bus import MessageBus
from src.workflows.langgraph_workflow import StateGraph
from src.monitoring.langsmith_integration import EnterpriseTracing

print('✓ All modules imported successfully')
print(f'✓ Python version: {sys.version}')
print(f'✓ Environment: {os.getenv(\"ENVIRONMENT\", \"development\")}')
"

# 数据库迁移
echo "Running database migrations..."
python -c "
# from src.database.migrations import run_migrations  # 示例代码,实际项目中需根据数据库方案调整
run_migrations()
print('✓ Database migrations completed')
"

# 预热系统
echo "Warming up system..."
python -c "
import asyncio
from main import MultiAgentSystem

async def warmup():
    system = MultiAgentSystem()
    await system.initialize()
    print('✓ System warmup completed')
    await system.shutdown()

asyncio.run(warmup())
"

# 启动主应用
echo "Starting main application..."
exec python main.py
"""

1.2 核心智能体实现

1.2.1 对话管理智能体

from langgraph import StateGraph, END
from langchain.schema import BaseMessage
from typing import Dict, List, Any
import asyncio

class DialogManagerAgent:
    """对话管理智能体"""

    def __init__(self):
        self.conversation_history = {}
        self.context_window = 10

    async def process_message(self, user_id: str, message: str) -> Dict[str, Any]:
        """处理用户消息"""
        # 获取对话历史
        history = self.get_conversation_history(user_id)

        # 构建状态图
        workflow = StateGraph(ConversationState)

        # 添加节点
        workflow.add_node("intent_recognition", self.recognize_intent)
        workflow.add_node("knowledge_retrieval", self.retrieve_knowledge)
        workflow.add_node("response_generation", self.generate_response)
        workflow.add_node("sentiment_analysis", self.analyze_sentiment)

        # 定义边
        workflow.add_edge("intent_recognition", "knowledge_retrieval")
        workflow.add_edge("knowledge_retrieval", "response_generation")
        workflow.add_edge("response_generation", "sentiment_analysis")
        workflow.add_edge("sentiment_analysis", END)

        # 设置入口点
        workflow.set_entry_point("intent_recognition")

        # 编译并运行
        app = workflow.compile()

        initial_state = ConversationState(
            user_id=user_id,
            message=message,
            history=history,
            context={}
        )

        result = await app.ainvoke(initial_state)
        return result

class ConversationState:
    """对话状态"""
    def __init__(self, user_id: str, message: str, history: List, context: Dict):
        self.user_id = user_id
        self.message = message
        self.history = history
        self.context = context
        self.intent = None
        self.knowledge = None
        self.response = None
        self.sentiment = None

1.2.2 知识检索智能体

from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings
from langchain.text_splitter import RecursiveCharacterTextSplitter

class KnowledgeRetrieverAgent:
    """知识检索智能体"""

    def __init__(self):
        self.embeddings = OpenAIEmbeddings()
        self.vectorstore = None
        self.setup_knowledge_base()

    def setup_knowledge_base(self):
        """设置知识库"""
        # 加载企业知识文档
        documents = self.load_knowledge_documents()

        # 文档分割
        text_splitter = RecursiveCharacterTextSplitter(
            chunk_size=1000,
            chunk_overlap=200
        )
        splits = text_splitter.split_documents(documents)

        # 创建向量存储
        self.vectorstore = Chroma.from_documents(
            documents=splits,
            embedding=self.embeddings,
            persist_directory="./knowledge_db"
        )

    async def retrieve_knowledge(self, query: str, top_k: int = 5) -> List[Dict]:
        """检索相关知识"""
        # 相似性搜索
        docs = self.vectorstore.similarity_search_with_score(query, k=top_k)

        # 重排序
        reranked_docs = await self.rerank_documents(query, docs)

        return [
            {
                "content": doc.page_content,
                "metadata": doc.metadata,
                "score": score
            }
            for doc, score in reranked_docs
        ]

    async def rerank_documents(self, query: str, docs: List) -> List:
        """文档重排序"""
        # 使用更精确的重排序模型
        # 这里可以集成 Cohere Rerank 或其他重排序服务
        return docs  # 简化实现

1.3 系统集成与测试

1.3.1 集成测试框架

import pytest
import asyncio
from unittest.mock import Mock, patch

class IntegrationTestSuite:
    """集成测试套件"""

    @pytest.fixture
    async def customer_service_system(self):
        """客服系统测试夹具"""
        system = CustomerServiceSystem()
        await system.initialize()
        yield system
        await system.cleanup()

    @pytest.mark.asyncio
    async def test_end_to_end_conversation(self, customer_service_system):
        """端到端对话测试"""
        user_id = "test_user_001"

        # 测试场景1:产品咨询
        response1 = await customer_service_system.process_message(
            user_id, "我想了解你们的产品价格"
        )
        assert response1["intent"] == "product_inquiry"
        assert "价格" in response1["response"]

        # 测试场景2:技术支持
        response2 = await customer_service_system.process_message(
            user_id, "我的账号登录不了"
        )
        assert response2["intent"] == "technical_support"
        assert response2["escalate_to_human"] == False

        # 测试场景3:投诉处理
        response3 = await customer_service_system.process_message(
            user_id, "你们的服务太差了,我要投诉"
        )
        assert response3["sentiment"] == "negative"
        assert response3["escalate_to_human"] == True

    @pytest.mark.asyncio
    async def test_performance_under_load(self, customer_service_system):
        """负载测试"""
        # 模拟100个并发用户
        tasks = []
        for i in range(100):
            task = customer_service_system.process_message(
                f"user_{i}", "Hello, I need help"
            )
            tasks.append(task)

        # 执行并发测试
        start_time = asyncio.get_event_loop().time()
        results = await asyncio.gather(*tasks)
        end_time = asyncio.get_event_loop().time()

        # 验证性能指标
        avg_response_time = (end_time - start_time) / len(results)
        assert avg_response_time < 2.0  # 平均响应时间小于2秒
        assert all(result["status"] == "success" for result in results)

2. 生产环境部署

2.1 容器化部署

2.1.1 Docker配置

# Dockerfile
FROM python:3.11-slim

WORKDIR /app

# 安装系统依赖
RUN apt-get update && apt-get install -y \
    gcc \
    g++ \
    && rm -rf /var/lib/apt/lists/*

# 复制依赖文件
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# 复制应用代码
COPY . .

# 设置环境变量
ENV PYTHONPATH=/app
ENV LANGCHAIN_TRACING_V2=true
ENV LANGCHAIN_ENDPOINT=https://api.smith.langchain.com

# 暴露端口
EXPOSE 8000

# 健康检查
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
    CMD curl -f http://localhost:8000/health || exit 1

# 启动命令
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

2.1.2 Docker Compose配置

# docker-compose.yml
version: '3.8'

services:
  customer-service:
    build: .
    ports:
      - "8000:8000"
    environment:
      - DATABASE_URL=postgresql://user:password@postgres:5432/customer_service
      - REDIS_URL=redis://redis:6379
      - ELASTICSEARCH_URL=http://elasticsearch:9200
    depends_on:
      - postgres
      - redis
      - elasticsearch
    volumes:
      - ./logs:/app/logs
    restart: unless-stopped

  postgres:
    image: postgres:15
    environment:
      POSTGRES_DB: customer_service
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password
    volumes:
      - postgres_data:/var/lib/postgresql/data
    ports:
      - "5432:5432"

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data

  elasticsearch:
    image: elasticsearch:8.8.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
    ports:
      - "9200:9200"
    volumes:
      - elasticsearch_data:/usr/share/elasticsearch/data

  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
      - ./ssl:/etc/nginx/ssl
    depends_on:
      - customer-service

volumes:
  postgres_data:
  redis_data:
  elasticsearch_data:

2.2 Kubernetes部署

2.2.1 K8s部署配置

# k8s-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: customer-service
  labels:
    app: customer-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: customer-service
  template:
    metadata:
      labels:
        app: customer-service
    spec:
      containers:
      - name: customer-service
        image: customer-service:latest
        ports:
        - containerPort: 8000
        env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: db-secret
              key: url
        - name: REDIS_URL
          value: "redis://redis-service:6379"
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "1Gi"
            cpu: "500m"
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 8000
          initialDelaySeconds: 5
          periodSeconds: 5

---
apiVersion: v1
kind: Service
metadata:
  name: customer-service
spec:
  selector:
    app: customer-service
  ports:
  - port: 80
    targetPort: 8000
  type: LoadBalancer

---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: customer-service-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: customer-service
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

3. 企业级监控与运维

3.1 基于实际项目的监控系统

3.1.1 企业级监控架构

# 基于项目 multi_agent_system/src/monitoring/langsmith_integration.py 的监控实现
import asyncio
import time
import psutil
import logging
from typing import Dict, List, Any, Optional
from dataclasses import dataclass, asdict
from datetime import datetime, timedelta
from prometheus_client import Counter, Histogram, Gauge, start_http_server

@dataclass
class SystemMetrics:
    """系统指标"""
    timestamp: datetime
    cpu_usage: float
    memory_usage: float
    disk_usage: float
    network_io: Dict[str, int]
    active_connections: int
    response_time_p95: float
    error_rate: float
    throughput: float

@dataclass
class AgentMetrics:
    """智能体指标"""
    agent_id: str
    agent_type: str
    total_requests: int
    successful_requests: int
    failed_requests: int
    avg_response_time: float
    current_load: float
    memory_usage: float
    last_activity: datetime

class EnterpriseMonitoringSystem:
    """企业级监控系统(基于实际项目)"""

    def __init__(self):
        self.setup_prometheus_metrics()
        self.setup_logging()
        self.alert_manager = AlertManager()
        self.performance_analyzer = PerformanceAnalyzer()

    def setup_prometheus_metrics(self):
        """设置 Prometheus 指标"""
        # 请求指标
        self.request_counter = Counter(
            'multiagent_requests_total',
            'Total number of requests',
            ['agent_type', 'status']
        )

        self.request_duration = Histogram(
            'multiagent_request_duration_seconds',
            'Request duration in seconds',
            ['agent_type', 'endpoint']
        )

        # 系统指标
        self.cpu_usage = Gauge(
            'multiagent_cpu_usage_percent',
            'CPU usage percentage'
        )

        self.memory_usage = Gauge(
            'multiagent_memory_usage_bytes',
            'Memory usage in bytes'
        )

        self.active_agents = Gauge(
            'multiagent_active_agents',
            'Number of active agents',
            ['agent_type']
        )

        # 业务指标
        self.customer_satisfaction = Gauge(
            'multiagent_customer_satisfaction_score',
            'Customer satisfaction score'
        )

        self.ticket_resolution_time = Histogram(
            'multiagent_ticket_resolution_seconds',
            'Ticket resolution time in seconds',
            ['priority', 'category']
        )

    def setup_logging(self):
        """设置日志系统"""
        logging.basicConfig(
            level=logging.INFO,
            format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
            handlers=[
                logging.FileHandler('/app/logs/multiagent.log'),
                logging.StreamHandler()
            ]
        )
        self.logger = logging.getLogger(__name__)

    async def collect_system_metrics(self) -> SystemMetrics:
        """收集系统指标"""
        # CPU 使用率
        cpu_usage = psutil.cpu_percent(interval=1)

        # 内存使用率
        memory = psutil.virtual_memory()
        memory_usage = memory.percent

        # 磁盘使用率
        disk = psutil.disk_usage('/')
        disk_usage = disk.percent

        # 网络IO
        network = psutil.net_io_counters()
        network_io = {
            'bytes_sent': network.bytes_sent,
            'bytes_recv': network.bytes_recv
        }

        # 活跃连接数
        connections = len(psutil.net_connections())

        # 更新 Prometheus 指标
        self.cpu_usage.set(cpu_usage)
        self.memory_usage.set(memory.used)

        return SystemMetrics(
            timestamp=datetime.now(),
            cpu_usage=cpu_usage,
            memory_usage=memory_usage,
            disk_usage=disk_usage,
            network_io=network_io,
            active_connections=connections,
            response_time_p95=0.0,  # 从应用指标获取
            error_rate=0.0,         # 从应用指标获取
            throughput=0.0          # 从应用指标获取
        )

    async def monitor_agent_performance(self, agent_id: str) -> AgentMetrics:
        """监控智能体性能"""
        # 从实际项目的智能体获取指标
        agent_stats = await self.get_agent_statistics(agent_id)

        metrics = AgentMetrics(
            agent_id=agent_id,
            agent_type=agent_stats.get('type', 'unknown'),
            total_requests=agent_stats.get('total_requests', 0),
            successful_requests=agent_stats.get('successful_requests', 0),
            failed_requests=agent_stats.get('failed_requests', 0),
            avg_response_time=agent_stats.get('avg_response_time', 0.0),
            current_load=agent_stats.get('current_load', 0.0),
            memory_usage=agent_stats.get('memory_usage', 0.0),
            last_activity=datetime.now()
        )

        # 更新 Prometheus 指标
        self.active_agents.labels(agent_type=metrics.agent_type).set(1)

        return metrics

class AlertManager:
    """告警管理器"""

    def __init__(self):
        self.alert_rules = self.load_alert_rules()
        self.notification_channels = self.setup_notification_channels()

    def load_alert_rules(self) -> List[Dict]:
        """加载告警规则"""
        return [
            {
                "name": "high_cpu_usage",
                "condition": "cpu_usage > 80",
                "severity": "warning",
                "duration": "5m",
                "message": "CPU usage is above 80% for 5 minutes"
            },
            {
                "name": "high_memory_usage", 
                "condition": "memory_usage > 90",
                "severity": "critical",
                "duration": "2m",
                "message": "Memory usage is above 90% for 2 minutes"
            },
            {
                "name": "high_error_rate",
                "condition": "error_rate > 5",
                "severity": "critical", 
                "duration": "1m",
                "message": "Error rate is above 5% for 1 minute"
            },
            {
                "name": "slow_response_time",
                "condition": "response_time_p95 > 5",
                "severity": "warning",
                "duration": "3m", 
                "message": "95th percentile response time is above 5 seconds"
            },
            {
                "name": "agent_down",
                "condition": "active_agents == 0",
                "severity": "critical",
                "duration": "30s",
                "message": "No active agents detected"
            }
        ]

class PerformanceAnalyzer:
    """性能分析器"""

    def __init__(self):
        self.metrics_history = []
        self.analysis_window = timedelta(hours=1)

    async def analyze_performance_trends(self, metrics: SystemMetrics) -> Dict[str, Any]:
        """分析性能趋势"""
        self.metrics_history.append(metrics)

        # 保持分析窗口大小
        cutoff_time = datetime.now() - self.analysis_window
        self.metrics_history = [
            m for m in self.metrics_history 
            if m.timestamp > cutoff_time
        ]

        if len(self.metrics_history) < 10:
            return {"status": "insufficient_data"}

        # 计算趋势
        cpu_trend = self.calculate_trend([m.cpu_usage for m in self.metrics_history])
        memory_trend = self.calculate_trend([m.memory_usage for m in self.metrics_history])
        response_time_trend = self.calculate_trend([m.response_time_p95 for m in self.metrics_history])

        # 异常检测
        anomalies = self.detect_anomalies()

        # 性能建议
        recommendations = self.generate_recommendations(cpu_trend, memory_trend, response_time_trend)

        return {
            "status": "analysis_complete",
            "trends": {
                "cpu": cpu_trend,
                "memory": memory_trend,
                "response_time": response_time_trend
            },
            "anomalies": anomalies,
            "recommendations": recommendations,
            "analysis_period": {
                "start": self.metrics_history[0].timestamp.isoformat(),
                "end": self.metrics_history[-1].timestamp.isoformat(),
                "sample_count": len(self.metrics_history)
            }
        }

# 生产环境监控配置
class ProductionMonitoringConfig:
    """生产环境监控配置"""

    @staticmethod
    def get_prometheus_config():
        """Prometheus 配置"""
        return """
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "alert_rules.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - alertmanager:9093

scrape_configs:
  - job_name: 'multi-agent-system'
    static_configs:
      - targets: ['multi-agent-system:8000']
    metrics_path: '/metrics'
    scrape_interval: 10s

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']

  - job_name: 'postgres-exporter'
    static_configs:
      - targets: ['postgres-exporter:9187']

  - job_name: 'redis-exporter'
    static_configs:
      - targets: ['redis-exporter:9121']
"""

    @staticmethod
    def get_grafana_dashboard():
        """Grafana 仪表板配置"""
        return {
            "dashboard": {
                "title": "Multi-Agent System Monitoring",
                "panels": [
                    {
                        "title": "System Overview",
                        "type": "stat",
                        "targets": [
                            {"expr": "multiagent_cpu_usage_percent"},
                            {"expr": "multiagent_memory_usage_bytes / 1024 / 1024 / 1024"},
                            {"expr": "rate(multiagent_requests_total[5m])"}
                        ]
                    },
                    {
                        "title": "Request Rate",
                        "type": "graph",
                        "targets": [
                            {"expr": "rate(multiagent_requests_total[5m])"}
                        ]
                    },
                    {
                        "title": "Response Time",
                        "type": "graph", 
                        "targets": [
                            {"expr": "histogram_quantile(0.95, multiagent_request_duration_seconds_bucket)"},
                            {"expr": "histogram_quantile(0.50, multiagent_request_duration_seconds_bucket)"}
                        ]
                    },
                    {
                        "title": "Error Rate",
                        "type": "graph",
                        "targets": [
                            {"expr": "rate(multiagent_requests_total{status=\"error\"}[5m]) / rate(multiagent_requests_total[5m]) * 100"}
                        ]
                    },
                    {
                        "title": "Active Agents",
                        "type": "stat",
                        "targets": [
                            {"expr": "multiagent_active_agents"}
                        ]
                    },
                    {
                        "title": "Customer Satisfaction",
                        "type": "gauge",
                        "targets": [
                            {"expr": "multiagent_customer_satisfaction_score"}
                        ]
                    }
                ]
            }
        }

# 启动监控系统
async def start_monitoring_system():
    """启动监控系统"""
    monitoring = EnterpriseMonitoringSystem()

    # 启动 Prometheus 指标服务器
    start_http_server(8001)

    # 监控循环
    while True:
        try:
            # 收集系统指标
            system_metrics = await monitoring.collect_system_metrics()

            # 检查告警
            await monitoring.alert_manager.check_alerts(system_metrics)

            # 性能分析
            analysis = await monitoring.performance_analyzer.analyze_performance_trends(system_metrics)

            # 记录日志
            monitoring.logger.info(f"System metrics collected: {system_metrics}")

            if analysis.get("status") == "analysis_complete":
                monitoring.logger.info(f"Performance analysis: {analysis}")

            # 等待下一次收集
            await asyncio.sleep(30)

        except Exception as e:
            monitoring.logger.error(f"Monitoring error: {e}")
            await asyncio.sleep(60)

3.2 自动化运维实践

3.2.1 自动扩缩容

# 基于实际项目的自动扩缩容实现
import kubernetes
from kubernetes import client, config
import asyncio
import logging
from typing import Dict, List, Any
from dataclasses import dataclass
from datetime import datetime, timedelta

@dataclass
class ScalingMetrics:
    """扩缩容指标"""
    cpu_utilization: float
    memory_utilization: float
    request_rate: float
    response_time_p95: float
    error_rate: float
    queue_length: int
    active_connections: int

@dataclass
class ScalingDecision:
    """扩缩容决策"""
    action: str  # "scale_up", "scale_down", "no_action"
    target_replicas: int
    reason: str
    confidence: float
    timestamp: datetime

class AutoScaler:
    """自动扩缩容器"""

    def __init__(self):
        self.setup_kubernetes_client()
        self.scaling_config = self.load_scaling_config()
        self.metrics_history = []
        self.logger = logging.getLogger(__name__)

    def setup_kubernetes_client(self):
        """设置 Kubernetes 客户端"""
        try:
            # 尝试集群内配置
            config.load_incluster_config()
        except:
            # 使用本地配置
            config.load_kube_config()

        self.apps_v1 = client.AppsV1Api()
        self.core_v1 = client.CoreV1Api()

    def load_scaling_config(self) -> Dict[str, Any]:
        """加载扩缩容配置"""
        return {
            "deployment_name": "multi-agent-system",
            "namespace": "default",
            "min_replicas": 2,
            "max_replicas": 20,
            "target_cpu_utilization": 70,
            "target_memory_utilization": 80,
            "target_response_time": 2.0,
            "scale_up_threshold": {
                "cpu": 80,
                "memory": 85,
                "response_time": 3.0,
                "error_rate": 5.0
            },
            "scale_down_threshold": {
                "cpu": 30,
                "memory": 40,
                "response_time": 1.0,
                "error_rate": 1.0
            },
            "cooldown_period": 300,  # 5分钟冷却期
            "scale_up_factor": 1.5,
            "scale_down_factor": 0.8
        }

    async def make_scaling_decision(self, metrics: ScalingMetrics) -> ScalingDecision:
        """制定扩缩容决策"""
        current_replicas = await self.get_current_replicas()

        # 检查冷却期
        if not self.is_cooldown_expired():
            return ScalingDecision(
                action="no_action",
                target_replicas=current_replicas,
                reason="在冷却期内",
                confidence=1.0,
                timestamp=datetime.now()
            )

        # 扩容条件检查
        scale_up_score = self.calculate_scale_up_score(metrics)
        scale_down_score = self.calculate_scale_down_score(metrics)

        if scale_up_score > 0.7:
            target_replicas = min(
                int(current_replicas * self.scaling_config["scale_up_factor"]),
                self.scaling_config["max_replicas"]
            )
            return ScalingDecision(
                action="scale_up",
                target_replicas=target_replicas,
                reason=f"扩容评分: {scale_up_score:.2f}",
                confidence=scale_up_score,
                timestamp=datetime.now()
            )

        elif scale_down_score > 0.7:
            target_replicas = max(
                int(current_replicas * self.scaling_config["scale_down_factor"]),
                self.scaling_config["min_replicas"]
            )
            return ScalingDecision(
                action="scale_down",
                target_replicas=target_replicas,
                reason=f"缩容评分: {scale_down_score:.2f}",
                confidence=scale_down_score,
                timestamp=datetime.now()
            )

        return ScalingDecision(
            action="no_action",
            target_replicas=current_replicas,
            reason="指标正常",
            confidence=0.5,
            timestamp=datetime.now()
        )

# 自动化运维主循环
async def auto_scaling_loop():
    """自动扩缩容主循环"""
    autoscaler = AutoScaler()

    while True:
        try:
            # 收集指标
            metrics = await autoscaler.collect_scaling_metrics()

            # 制定决策
            decision = await autoscaler.make_scaling_decision(metrics)

            # 执行扩缩容
            success = await autoscaler.execute_scaling(decision)

            if success:
                autoscaler.logger.info(f"扩缩容决策: {decision}")

            # 等待下一次检查
            await asyncio.sleep(60)  # 每分钟检查一次

        except Exception as e:
            autoscaler.logger.error(f"自动扩缩容错误: {e}")
            await asyncio.sleep(300)  # 错误时等待5分钟

3.3 故障恢复与灾备

3.3.1 故障检测与自动恢复

# 基于实际项目的故障恢复机制
import asyncio
import logging
from typing import Dict, List, Any, Optional
from dataclasses import dataclass
from datetime import datetime, timedelta
from enum import Enum

class FailureType(Enum):
    """故障类型"""
    SERVICE_DOWN = "service_down"
    HIGH_ERROR_RATE = "high_error_rate"
    SLOW_RESPONSE = "slow_response"
    MEMORY_LEAK = "memory_leak"
    DATABASE_CONNECTION = "database_connection"
    NETWORK_ISSUE = "network_issue"

class RecoveryAction(Enum):
    """恢复动作"""
    RESTART_SERVICE = "restart_service"
    SCALE_UP = "scale_up"
    FAILOVER = "failover"
    CIRCUIT_BREAKER = "circuit_breaker"
    ROLLBACK = "rollback"
    MANUAL_INTERVENTION = "manual_intervention"

@dataclass
class FailureEvent:
    """故障事件"""
    failure_type: FailureType
    severity: str
    description: str
    affected_services: List[str]
    timestamp: datetime
    metrics: Dict[str, Any]
    recovery_actions: List[RecoveryAction]

class FailureDetector:
    """故障检测器"""

    def __init__(self):
        self.detection_rules = self.load_detection_rules()
        self.logger = logging.getLogger(__name__)

    def load_detection_rules(self) -> Dict[FailureType, Dict]:
        """加载故障检测规则"""
        return {
            FailureType.SERVICE_DOWN: {
                "condition": "health_check_failures > 3",
                "window": "2m",
                "severity": "critical"
            },
            FailureType.HIGH_ERROR_RATE: {
                "condition": "error_rate > 10",
                "window": "5m",
                "severity": "high"
            },
            FailureType.SLOW_RESPONSE: {
                "condition": "response_time_p95 > 10",
                "window": "5m",
                "severity": "medium"
            },
            FailureType.MEMORY_LEAK: {
                "condition": "memory_growth_rate > 5",
                "window": "30m",
                "severity": "high"
            }
        }

    async def detect_failures(self, metrics: Dict[str, Any]) -> List[FailureEvent]:
        """检测故障"""
        failures = []

        for failure_type, rule in self.detection_rules.items():
            if self.evaluate_condition(rule["condition"], metrics):
                failure = FailureEvent(
                    failure_type=failure_type,
                    severity=rule["severity"],
                    description=f"检测到{failure_type.value}故障",
                    affected_services=self.identify_affected_services(failure_type, metrics),
                    timestamp=datetime.now(),
                    metrics=metrics,
                    recovery_actions=self.determine_recovery_actions(failure_type)
                )
                failures.append(failure)

        return failures

    def determine_recovery_actions(self, failure_type: FailureType) -> List[RecoveryAction]:
        """确定恢复动作"""
        recovery_map = {
            FailureType.SERVICE_DOWN: [RecoveryAction.RESTART_SERVICE, RecoveryAction.FAILOVER],
            FailureType.HIGH_ERROR_RATE: [RecoveryAction.CIRCUIT_BREAKER, RecoveryAction.ROLLBACK],
            FailureType.SLOW_RESPONSE: [RecoveryAction.SCALE_UP, RecoveryAction.CIRCUIT_BREAKER],
            FailureType.MEMORY_LEAK: [RecoveryAction.RESTART_SERVICE],
            FailureType.DATABASE_CONNECTION: [RecoveryAction.FAILOVER, RecoveryAction.RESTART_SERVICE],
            FailureType.NETWORK_ISSUE: [RecoveryAction.FAILOVER, RecoveryAction.MANUAL_INTERVENTION]
        }
        return recovery_map.get(failure_type, [RecoveryAction.MANUAL_INTERVENTION])

class AutoRecoverySystem:
    """自动恢复系统"""

    def __init__(self):
        self.failure_detector = FailureDetector()
        self.recovery_executor = RecoveryExecutor()
        self.logger = logging.getLogger(__name__)

    async def monitor_and_recover(self):
        """监控和恢复主循环"""
        while True:
            try:
                # 收集系统指标
                metrics = await self.collect_system_metrics()

                # 检测故障
                failures = await self.failure_detector.detect_failures(metrics)

                # 执行恢复动作
                for failure in failures:
                    await self.handle_failure(failure)

                await asyncio.sleep(30)

            except Exception as e:
                self.logger.error(f"监控恢复系统错误: {e}")
                await asyncio.sleep(60)

    async def handle_failure(self, failure: FailureEvent):
        """处理故障"""
        self.logger.warning(f"检测到故障: {failure}")

        for action in failure.recovery_actions:
            try:
                success = await self.recovery_executor.execute_action(action, failure)
                if success:
                    self.logger.info(f"恢复动作 {action} 执行成功")
                    break
                else:
                    self.logger.warning(f"恢复动作 {action} 执行失败")
            except Exception as e:
                self.logger.error(f"执行恢复动作 {action} 时出错: {e}")

        # 发送告警通知
        await self.send_failure_notification(failure)

class RecoveryExecutor:
    """恢复执行器"""

    async def execute_action(self, action: RecoveryAction, failure: FailureEvent) -> bool:
        """执行恢复动作"""
        if action == RecoveryAction.RESTART_SERVICE:
            return await self.restart_service(failure.affected_services)
        elif action == RecoveryAction.SCALE_UP:
            return await self.scale_up_service(failure.affected_services)
        elif action == RecoveryAction.FAILOVER:
            return await self.failover_service(failure.affected_services)
        elif action == RecoveryAction.CIRCUIT_BREAKER:
            return await self.enable_circuit_breaker(failure.affected_services)
        elif action == RecoveryAction.ROLLBACK:
            return await self.rollback_deployment(failure.affected_services)
        else:
            return False

    async def restart_service(self, services: List[str]) -> bool:
        """重启服务"""
        try:
            for service in services:
                # 使用 Kubernetes API 重启服务
                await self.kubernetes_restart_deployment(service)
            return True
        except Exception as e:
            logging.error(f"重启服务失败: {e}")
            return False

    async def kubernetes_restart_deployment(self, deployment_name: str):
        """Kubernetes 重启部署"""
        # 实际的 Kubernetes API 调用
        pass

3.4 性能优化实践

3.4.1 系统性能调优

# 基于实际项目的性能优化实现
import asyncio
import cProfile
import pstats
import io
import logging
from typing import Dict, List, Any, Optional
from dataclasses import dataclass
from datetime import datetime
import psutil
import gc
from prometheus_client import Counter, Gauge, Histogram

# Prometheus 指标定义
REQUEST_COUNT = Counter('requests_total', 'Total requests', ['method', 'endpoint'])
ACTIVE_CONVERSATIONS = Gauge('active_conversations', 'Number of active conversations')
RESPONSE_TIME = Histogram('response_time_seconds', 'Response time in seconds')

def monitor_performance(operation_name: str):
    """性能监控装饰器"""
    def decorator(func):
        async def wrapper(*args, **kwargs):
            start_time = datetime.now()
            try:
                result = await func(*args, **kwargs)
                return result
            finally:
                duration = (datetime.now() - start_time).total_seconds()
                RESPONSE_TIME.observe(duration)
                logging.info(f"{operation_name} 执行时间: {duration:.2f}秒")
        return wrapper
    return decorator

@dataclass
class PerformanceProfile:
    """性能分析结果"""
    function_stats: Dict[str, Any]
    memory_usage: Dict[str, float]
    cpu_usage: float
    io_stats: Dict[str, int]
    bottlenecks: List[str]
    recommendations: List[str]

class PerformanceOptimizer:
    """性能优化器"""

    def __init__(self):
        self.profiler = cProfile.Profile()
        self.optimization_history = []
        self.logger = logging.getLogger(__name__)

    async def profile_system_performance(self) -> PerformanceProfile:
        """分析系统性能"""
        # 开始性能分析
        self.profiler.enable()

        # 运行一段时间收集数据
        await asyncio.sleep(60)

        # 停止分析
        self.profiler.disable()

        # 分析结果
        s = io.StringIO()
        ps = pstats.Stats(self.profiler, stream=s)
        ps.sort_stats('cumulative')
        ps.print_stats()

        # 收集系统指标
        memory_info = psutil.virtual_memory()
        cpu_usage = psutil.cpu_percent()
        io_counters = psutil.disk_io_counters()

        # 识别瓶颈
        bottlenecks = self.identify_bottlenecks(ps)

        # 生成优化建议
        recommendations = self.generate_optimization_recommendations(bottlenecks)

        return PerformanceProfile(
            function_stats=self.extract_function_stats(ps),
            memory_usage={
                "total": memory_info.total,
                "used": memory_info.used,
                "percent": memory_info.percent
            },
            cpu_usage=cpu_usage,
            io_stats={
                "read_bytes": io_counters.read_bytes,
                "write_bytes": io_counters.write_bytes
            },
            bottlenecks=bottlenecks,
            recommendations=recommendations
        )

    def identify_bottlenecks(self, stats: pstats.Stats) -> List[str]:
        """识别性能瓶颈"""
        bottlenecks = []

        # 分析最耗时的函数
        stats.sort_stats('cumulative')
        top_functions = stats.get_stats_profile().func_profiles

        for func, (cc, nc, tt, ct, callers) in list(top_functions.items())[:10]:
            if ct > 1.0:  # 累计时间超过1秒
                bottlenecks.append(f"函数 {func} 耗时过长: {ct:.2f}秒")

        return bottlenecks

    def generate_optimization_recommendations(self, bottlenecks: List[str]) -> List[str]:
        """生成优化建议"""
        recommendations = []

        for bottleneck in bottlenecks:
            if "数据库" in bottleneck:
                recommendations.append("优化数据库查询,添加索引或使用缓存")
            elif "网络" in bottleneck:
                recommendations.append("优化网络请求,使用连接池或异步处理")
            elif "内存" in bottleneck:
                recommendations.append("优化内存使用,检查内存泄漏")
            elif "CPU" in bottleneck:
                recommendations.append("优化算法复杂度,使用并行处理")
            else:
                recommendations.append("进一步分析具体瓶颈原因")

        return recommendations

    async def apply_optimizations(self, profile: PerformanceProfile):
        """应用性能优化"""
        for recommendation in profile.recommendations:
            if "缓存" in recommendation:
                await self.optimize_caching()
            elif "数据库" in recommendation:
                await self.optimize_database()
            elif "内存" in recommendation:
                await self.optimize_memory()
            elif "并行" in recommendation:
                await self.optimize_concurrency()

    async def optimize_caching(self):
        """优化缓存策略"""
        # 实现缓存优化逻辑
        self.logger.info("应用缓存优化策略")

    async def optimize_database(self):
        """优化数据库性能"""
        # 实现数据库优化逻辑
        self.logger.info("应用数据库优化策略")

    async def optimize_memory(self):
        """优化内存使用"""
        # 强制垃圾回收
        gc.collect()
        self.logger.info("执行内存优化")

    async def optimize_concurrency(self):
        """优化并发处理"""
        # 实现并发优化逻辑
        self.logger.info("应用并发优化策略")

    def extract_function_stats(self, stats: pstats.Stats) -> Dict[str, Any]:
        """提取函数统计信息"""
        function_stats = {}
        stats_profile = stats.get_stats_profile()

        for func, (cc, nc, tt, ct, callers) in stats_profile.func_profiles.items():
            function_stats[str(func)] = {
                "call_count": cc,
                "total_time": tt,
                "cumulative_time": ct,
                "per_call_time": tt / cc if cc > 0 else 0
            }

        return function_stats

# 性能监控和优化主循环
async def performance_optimization_loop():
    """性能优化主循环"""
    optimizer = PerformanceOptimizer()

    while True:
        try:
            # 分析性能
            profile = await optimizer.profile_system_performance()

            # 记录分析结果
            optimizer.logger.info(f"性能分析完成: {profile}")

            # 应用优化
            await optimizer.apply_optimizations(profile)

            # 等待下一次分析
            await asyncio.sleep(3600)  # 每小时分析一次

        except Exception as e:
            optimizer.logger.error(f"性能优化错误: {e}")
            await asyncio.sleep(1800)  # 错误时等待30分钟

# 使用示例
async def run_performance_optimization():
    """运行性能优化示例"""
    optimizer = PerformanceOptimizer()

    # 启动性能优化循环
    await performance_optimization_loop()

# 带监控的对话管理器示例
class MonitoredDialogManager:
    """带监控的对话管理器"""

    def __init__(self):
        self.logger = logging.getLogger(__name__)

    @monitor_performance("dialog_manager")
    async def process_message(self, user_id: str, message: str):
        """处理用户消息"""
        ACTIVE_CONVERSATIONS.inc()
        try:
            # 模拟消息处理逻辑
            result = await self._handle_message(user_id, message)
            REQUEST_COUNT.labels(method='POST', endpoint='/chat').inc()
            return result
        finally:
            ACTIVE_CONVERSATIONS.dec()

    async def _handle_message(self, user_id: str, message: str) -> str:
        """处理消息的具体逻辑"""
        # 这里实现具体的消息处理逻辑
        await asyncio.sleep(0.1)  # 模拟处理时间
        return f"处理用户 {user_id} 的消息: {message}"

3.2 日志管理

3.2.1 结构化日志配置

import logging
import json
from datetime import datetime
from typing import Dict, Any

class StructuredLogger:
    """结构化日志记录器"""

    def __init__(self, name: str):
        self.logger = logging.getLogger(name)
        self.logger.setLevel(logging.INFO)

        # 配置处理器
        handler = logging.StreamHandler()
        formatter = logging.Formatter('%(message)s')
        handler.setFormatter(formatter)
        self.logger.addHandler(handler)

    def log_event(self, event_type: str, data: Dict[str, Any], level: str = "INFO"):
        """记录结构化事件"""
        log_entry = {
            "timestamp": datetime.utcnow().isoformat(),
            "event_type": event_type,
            "level": level,
            "data": data
        }

        if level == "ERROR":
            self.logger.error(json.dumps(log_entry))
        elif level == "WARNING":
            self.logger.warning(json.dumps(log_entry))
        else:
            self.logger.info(json.dumps(log_entry))

# 使用示例
logger = StructuredLogger("customer_service")

# 记录对话事件
logger.log_event("conversation_started", {
    "user_id": "user_123",
    "session_id": "session_456",
    "channel": "web"
})

# 记录错误事件
logger.log_event("agent_error", {
    "agent_type": "knowledge_retriever",
    "error_message": "Connection timeout",
    "user_id": "user_123"
}, level="ERROR")

3.3 性能优化

3.3.1 缓存策略

import redis
import pickle
from typing import Optional, Any
import hashlib

class CacheManager:
    """缓存管理器"""

    def __init__(self, redis_url: str):
        self.redis_client = redis.from_url(redis_url)
        self.default_ttl = 3600  # 1小时

    def _generate_key(self, prefix: str, *args) -> str:
        """生成缓存键"""
        key_data = f"{prefix}:{':'.join(map(str, args))}"
        return hashlib.md5(key_data.encode()).hexdigest()

    async def get(self, prefix: str, *args) -> Optional[Any]:
        """获取缓存"""
        key = self._generate_key(prefix, *args)
        data = self.redis_client.get(key)
        if data:
            return pickle.loads(data)
        return None

    async def set(self, prefix: str, value: Any, ttl: Optional[int] = None, *args):
        """设置缓存"""
        key = self._generate_key(prefix, *args)
        ttl = ttl or self.default_ttl
        self.redis_client.setex(key, ttl, pickle.dumps(value))

    def cache_result(self, prefix: str, ttl: Optional[int] = None):
        """缓存装饰器"""
        def decorator(func):
            @functools.wraps(func)
            async def wrapper(*args, **kwargs):
                # 生成缓存键
                cache_key = self._generate_key(prefix, *args, **kwargs)

                # 尝试从缓存获取
                cached_result = await self.get(prefix, *args, **kwargs)
                if cached_result is not None:
                    return cached_result

                # 执行函数并缓存结果
                result = await func(*args, **kwargs)
                await self.set(prefix, result, ttl, *args, **kwargs)
                return result
            return wrapper
        return decorator

# 应用缓存
cache_manager = CacheManager("redis://localhost:6379")

class CachedKnowledgeRetriever(KnowledgeRetrieverAgent):
    """带缓存的知识检索器"""

    @cache_manager.cache_result("knowledge_search", ttl=1800)
    async def retrieve_knowledge(self, query: str, top_k: int = 5):
        return await super().retrieve_knowledge(query, top_k)

4. 容器化部署与云原生运维

4.1 完整容器化部署方案

4.1.1 多阶段Dockerfile优化

# 基于实际项目的多阶段构建Dockerfile
FROM python:3.11-slim as builder

# 设置工作目录
WORKDIR /app

# 安装系统依赖
RUN apt-get update && apt-get install -y \
    gcc \
    g++ \
    && rm -rf /var/lib/apt/lists/*

# 复制依赖文件
COPY requirements.txt .

# 安装Python依赖
RUN pip install --no-cache-dir --user -r requirements.txt

# 生产阶段
FROM python:3.11-slim as production

# 创建非root用户
RUN groupadd -r appuser && useradd -r -g appuser appuser

# 设置工作目录
WORKDIR /app

# 从builder阶段复制依赖
COPY --from=builder /root/.local /home/appuser/.local

# 复制应用代码
COPY --chown=appuser:appuser . .

# 设置环境变量
ENV PATH=/home/appuser/.local/bin:$PATH
ENV PYTHONPATH=/app
ENV PYTHONUNBUFFERED=1

# 切换到非root用户
USER appuser

# 健康检查
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
    CMD python -c "import requests; requests.get('http://localhost:8000/health')"

# 暴露端口
EXPOSE 8000

# 启动命令
CMD ["python", "main.py"]

4.1.2 生产级Docker Compose配置

# docker-compose.prod.yml - 生产环境配置
version: '3.8'

services:
  # 主应用服务
  customer-service:
    build:
      context: .
      dockerfile: Dockerfile
      target: production
    image: customer-service:latest
    container_name: customer-service-app
    restart: unless-stopped
    environment:
      - ENVIRONMENT=production
      - REDIS_URL=redis://redis:6379
      - DATABASE_URL=postgresql://user:password@postgres:5432/customer_service
      - LANGSMITH_API_KEY=${LANGSMITH_API_KEY}
      - OPENAI_API_KEY=${OPENAI_API_KEY}
    ports:
      - "8000:8000"
    depends_on:
      - redis
      - postgres
      - prometheus
    networks:
      - app-network
    volumes:
      - ./logs:/app/logs
      - ./config:/app/config:ro
    deploy:
      resources:
        limits:
          cpus: '2.0'
          memory: 4G
        reservations:
          cpus: '1.0'
          memory: 2G

  # Redis缓存服务
  redis:
    image: redis:7-alpine
    container_name: customer-service-redis
    restart: unless-stopped
    command: redis-server --appendonly yes --requirepass ${REDIS_PASSWORD}
    ports:
      - "6379:6379"
    volumes:
      - redis-data:/data
    networks:
      - app-network
    deploy:
      resources:
        limits:
          memory: 1G

  # PostgreSQL数据库
  postgres:
    image: postgres:15-alpine
    container_name: customer-service-db
    restart: unless-stopped
    environment:
      - POSTGRES_DB=customer_service
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
    ports:
      - "5432:5432"
    volumes:
      - postgres-data:/var/lib/postgresql/data
      - ./init.sql:/docker-entrypoint-initdb.d/init.sql:ro
    networks:
      - app-network
    deploy:
      resources:
        limits:
          memory: 2G

  # Nginx负载均衡器
  nginx:
    image: nginx:alpine
    container_name: customer-service-nginx
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - ./nginx/ssl:/etc/nginx/ssl:ro
      - ./logs/nginx:/var/log/nginx
    depends_on:
      - customer-service
    networks:
      - app-network

  # Prometheus监控
  prometheus:
    image: prom/prometheus:latest
    container_name: customer-service-prometheus
    restart: unless-stopped
    ports:
      - "9090:9090"
    volumes:
      - ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--storage.tsdb.retention.time=200h'
      - '--web.enable-lifecycle'
    networks:
      - app-network

  # Grafana仪表板
  grafana:
    image: grafana/grafana:latest
    container_name: customer-service-grafana
    restart: unless-stopped
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
    volumes:
      - grafana-data:/var/lib/grafana
      - ./monitoring/grafana/dashboards:/etc/grafana/provisioning/dashboards:ro
      - ./monitoring/grafana/datasources:/etc/grafana/provisioning/datasources:ro
    networks:
      - app-network

volumes:
  redis-data:
  postgres-data:
  prometheus-data:
  grafana-data:

networks:
  app-network:
    driver: bridge

4.2 Kubernetes云原生部署

4.2.1 Kubernetes部署清单

# k8s/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: customer-service
  labels:
    name: customer-service

---
# k8s/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: customer-service-config
  namespace: customer-service
data:
  ENVIRONMENT: "production"
  REDIS_URL: "redis://redis-service:6379"
  DATABASE_URL: "postgresql://user:password@postgres-service:5432/customer_service"

---
# k8s/secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: customer-service-secrets
  namespace: customer-service
type: Opaque
data:
  LANGSMITH_API_KEY: <base64-encoded-key>
  OPENAI_API_KEY: <base64-encoded-key>
  REDIS_PASSWORD: <base64-encoded-password>
  POSTGRES_PASSWORD: <base64-encoded-password>

---
# k8s/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: customer-service-app
  namespace: customer-service
  labels:
    app: customer-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: customer-service
  template:
    metadata:
      labels:
        app: customer-service
    spec:
      containers:
      - name: customer-service
        image: customer-service:latest
        ports:
        - containerPort: 8000
        envFrom:
        - configMapRef:
            name: customer-service-config
        - secretRef:
            name: customer-service-secrets
        resources:
          requests:
            memory: "1Gi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "1000m"
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 8000
          initialDelaySeconds: 5
          periodSeconds: 5
        volumeMounts:
        - name: logs
          mountPath: /app/logs
      volumes:
      - name: logs
        persistentVolumeClaim:
          claimName: customer-service-logs-pvc

---
# k8s/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: customer-service-svc
  namespace: customer-service
spec:
  selector:
    app: customer-service
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8000
  type: ClusterIP

---
# k8s/hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: customer-service-hpa
  namespace: customer-service
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: customer-service-app
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

4.2.2 Ingress和SSL配置

# k8s/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: customer-service-ingress
  namespace: customer-service
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/rate-limit: "100"
    nginx.ingress.kubernetes.io/rate-limit-window: "1m"
spec:
  tls:
  - hosts:
    - api.customer-service.com
    secretName: customer-service-tls
  rules:
  - host: api.customer-service.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: customer-service-svc
            port:
              number: 80

4.3 云原生运维自动化

4.3.1 GitOps部署流水线

# .github/workflows/deploy.yml
name: Deploy to Production

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3

    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.11'

    - name: Install dependencies
      run: |
        pip install -r requirements.txt
        pip install pytest pytest-cov

    - name: Run tests
      run: |
        pytest tests/ --cov=src/ --cov-report=xml

    - name: Upload coverage
      uses: codecov/codecov-action@v3

  build:
    needs: test
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    steps:
    - uses: actions/checkout@v3

    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v2

    - name: Login to Container Registry
      uses: docker/login-action@v2
      with:
        registry: ghcr.io
        username: ${{ github.actor }}
        password: ${{ secrets.GITHUB_TOKEN }}

    - name: Build and push
      uses: docker/build-push-action@v4
      with:
        context: .
        push: true
        tags: |
          ghcr.io/${{ github.repository }}:latest
          ghcr.io/${{ github.repository }}:${{ github.sha }}
        cache-from: type=gha
        cache-to: type=gha,mode=max

  deploy:
    needs: build
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    steps:
    - uses: actions/checkout@v3

    - name: Setup kubectl
      uses: azure/setup-kubectl@v3
      with:
        version: 'v1.28.0'

    - name: Deploy to Kubernetes
      run: |
        echo "${{ secrets.KUBECONFIG }}" | base64 -d > kubeconfig
        export KUBECONFIG=kubeconfig

        # 更新镜像标签
        kubectl set image deployment/customer-service-app \
          customer-service=ghcr.io/${{ github.repository }}:${{ github.sha }} \
          -n customer-service

        # 等待部署完成
        kubectl rollout status deployment/customer-service-app -n customer-service

4.3.2 自动化运维脚本

# scripts/auto_ops.py - 自动化运维脚本
import asyncio
import logging
from typing import Dict, List
from kubernetes import client, config
from prometheus_client.parser import text_string_to_metric_families
import requests

class CloudNativeOpsManager:
    """云原生运维管理器"""

    def __init__(self):
        # 加载Kubernetes配置
        try:
            config.load_incluster_config()  # 集群内配置
        except:
            config.load_kube_config()  # 本地配置

        self.k8s_apps_v1 = client.AppsV1Api()
        self.k8s_core_v1 = client.CoreV1Api()
        self.k8s_autoscaling_v2 = client.AutoscalingV2Api()

        self.logger = logging.getLogger(__name__)

        # 运维配置
        self.namespace = "customer-service"
        self.deployment_name = "customer-service-app"
        self.prometheus_url = "http://prometheus:9090"

    async def health_check(self) -> Dict[str, bool]:
        """系统健康检查"""
        health_status = {}

        try:
            # 检查Pod状态
            pods = self.k8s_core_v1.list_namespaced_pod(
                namespace=self.namespace,
                label_selector=f"app=customer-service"
            )

            healthy_pods = sum(1 for pod in pods.items 
                             if pod.status.phase == "Running")
            total_pods = len(pods.items)

            health_status["pods"] = healthy_pods == total_pods and total_pods > 0

            # 检查服务状态
            services = self.k8s_core_v1.list_namespaced_service(
                namespace=self.namespace
            )
            health_status["services"] = len(services.items) > 0

            # 检查应用响应
            try:
                response = requests.get(
                    f"http://customer-service-svc.{self.namespace}/health",
                    timeout=10
                )
                health_status["application"] = response.status_code == 200
            except:
                health_status["application"] = False

            self.logger.info(f"健康检查完成: {health_status}")
            return health_status

        except Exception as e:
            self.logger.error(f"健康检查失败: {e}")
            return {"error": True}

    async def auto_scale_decision(self) -> Dict[str, any]:
        """自动扩缩容决策"""
        try:
            # 获取当前HPA状态
            hpa = self.k8s_autoscaling_v2.read_namespaced_horizontal_pod_autoscaler(
                name="customer-service-hpa",
                namespace=self.namespace
            )

            current_replicas = hpa.status.current_replicas
            desired_replicas = hpa.status.desired_replicas

            # 获取性能指标
            metrics = await self.get_performance_metrics()

            # 扩缩容决策逻辑
            decision = {
                "current_replicas": current_replicas,
                "desired_replicas": desired_replicas,
                "cpu_usage": metrics.get("cpu_usage", 0),
                "memory_usage": metrics.get("memory_usage", 0),
                "request_rate": metrics.get("request_rate", 0),
                "action": "none"
            }

            # 基于业务指标的智能扩缩容
            if metrics.get("request_rate", 0) > 1000 and current_replicas < 8:
                decision["action"] = "scale_up"
                decision["reason"] = "High request rate detected"
            elif metrics.get("cpu_usage", 0) > 80 and current_replicas < 10:
                decision["action"] = "scale_up"
                decision["reason"] = "High CPU usage"
            elif (metrics.get("request_rate", 0) < 100 and 
                  metrics.get("cpu_usage", 0) < 30 and 
                  current_replicas > 3):
                decision["action"] = "scale_down"
                decision["reason"] = "Low resource usage"

            return decision

        except Exception as e:
            self.logger.error(f"扩缩容决策失败: {e}")
            return {"error": str(e)}

    async def get_performance_metrics(self) -> Dict[str, float]:
        """获取性能指标"""
        try:
            metrics = {}

            # CPU使用率
            cpu_query = f'avg(rate(container_cpu_usage_seconds_total{{namespace="{self.namespace}"}}[5m])) * 100'
            cpu_result = await self.query_prometheus(cpu_query)
            metrics["cpu_usage"] = float(cpu_result[0]["value"][1]) if cpu_result else 0

            # 内存使用率
            memory_query = f'avg(container_memory_usage_bytes{{namespace="{self.namespace}"}}) / avg(container_spec_memory_limit_bytes{{namespace="{self.namespace}"}}) * 100'
            memory_result = await self.query_prometheus(memory_query)
            metrics["memory_usage"] = float(memory_result[0]["value"][1]) if memory_result else 0

            # 请求速率
            request_query = f'sum(rate(http_requests_total{{namespace="{self.namespace}"}}[5m]))'
            request_result = await self.query_prometheus(request_query)
            metrics["request_rate"] = float(request_result[0]["value"][1]) if request_result else 0

            return metrics

        except Exception as e:
            self.logger.error(f"获取性能指标失败: {e}")
            return {}

    async def query_prometheus(self, query: str) -> List[Dict]:
        """查询Prometheus指标"""
        try:
            response = requests.get(
                f"{self.prometheus_url}/api/v1/query",
                params={"query": query},
                timeout=10
            )
            data = response.json()
            return data.get("data", {}).get("result", [])
        except Exception as e:
            self.logger.error(f"Prometheus查询失败: {e}")
            return []

    async def rolling_update(self, new_image: str):
        """滚动更新部署"""
        try:
            # 获取当前部署
            deployment = self.k8s_apps_v1.read_namespaced_deployment(
                name=self.deployment_name,
                namespace=self.namespace
            )

            # 更新镜像
            deployment.spec.template.spec.containers[0].image = new_image

            # 应用更新
            self.k8s_apps_v1.patch_namespaced_deployment(
                name=self.deployment_name,
                namespace=self.namespace,
                body=deployment
            )

            self.logger.info(f"开始滚动更新到镜像: {new_image}")

            # 等待更新完成
            await self.wait_for_rollout()

        except Exception as e:
            self.logger.error(f"滚动更新失败: {e}")
            raise

    async def wait_for_rollout(self, timeout: int = 600):
        """等待滚动更新完成"""
        import time
        start_time = time.time()

        while time.time() - start_time < timeout:
            try:
                deployment = self.k8s_apps_v1.read_namespaced_deployment(
                    name=self.deployment_name,
                    namespace=self.namespace
                )

                if (deployment.status.ready_replicas == deployment.spec.replicas and
                    deployment.status.updated_replicas == deployment.spec.replicas):
                    self.logger.info("滚动更新完成")
                    return True

                await asyncio.sleep(10)

            except Exception as e:
                self.logger.error(f"检查滚动更新状态失败: {e}")
                await asyncio.sleep(10)

        raise TimeoutError("滚动更新超时")

# 运维主循环
async def ops_main_loop():
    """运维主循环"""
    ops_manager = CloudNativeOpsManager()

    while True:
        try:
            # 健康检查
            health = await ops_manager.health_check()

            if not all(health.values()):
                logging.warning(f"系统健康检查异常: {health}")

            # 自动扩缩容决策
            scale_decision = await ops_manager.auto_scale_decision()

            if scale_decision.get("action") != "none":
                logging.info(f"扩缩容决策: {scale_decision}")

            # 等待下一次检查
            await asyncio.sleep(60)  # 每分钟检查一次

        except Exception as e:
            logging.error(f"运维循环错误: {e}")
            await asyncio.sleep(60)

if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)
    asyncio.run(ops_main_loop())

5. 实践练习

练习1:完整系统部署

  1. 使用Docker Compose部署完整的客服系统
  2. 配置Nginx负载均衡
  3. 设置SSL证书
  4. 验证系统功能

练习2:Kubernetes云原生部署

  1. 部署到Kubernetes集群
  2. 配置自动扩缩容
  3. 设置滚动更新
  4. 验证高可用性

练习3:监控告警配置

  1. 配置Prometheus监控指标
  2. 设置Grafana仪表板
  3. 配置告警规则
  4. 测试告警机制

练习4:性能优化实战

  1. 进行压力测试
  2. 识别性能瓶颈
  3. 实施优化方案
  4. 验证优化效果

课程总结

关键收获

  1. 完整开发流程:从需求分析到生产部署的全流程实践
  2. 企业级架构:可扩展、高可用的系统架构设计
  3. 运维最佳实践:监控、日志、性能优化的实战经验
  4. 问题解决能力:具备独立解决复杂技术问题的能力

后续学习建议

  1. 深入学习云原生技术(Kubernetes、Service Mesh)
  2. 探索更多AI技术(多模态、强化学习)
  3. 关注行业最新发展(AGI、具身智能)
  4. 参与开源项目,积累实战经验

认证考核

  • 完成综合项目实战
  • 通过技术答辩
  • 获得企业级多智能体系统开发认证

恭喜您完成了多智能体AI系统培训的全部课程!