第五天:应用实践与部署运维¶
学习目标¶
- 掌握多智能体系统的完整开发流程
- 学会企业级部署与运维最佳实践
- 具备独立构建多智能体系统的能力
- 了解生产环境的监控与优化策略
- 掌握容器化和云原生部署技术
参考项目¶
本课程将带领学员完成一个完整的企业级多智能体系统项目,从需求分析到部署运维的全流程实践。
💡 实际项目参考:本课程基于项目中的完整实现进行教学:
src/examples/customer_service_system.py- 完整的企业级客服系统实现main.py- 系统启动和部署脚本docker-compose.yml- 容器化部署配置docker-compose.yml- Kubernetes部署清单src/monitoring/- 监控和告警配置
实战特色:从实际企业需求出发,涵盖完整的开发、测试、部署、监控、运维流程,提供生产级的解决方案。
1. 综合项目实战¶
1.1 项目需求分析¶
1.1.1 基于实际项目的智能客服系统¶
**代码引用**: 完整的智能客服系统实现请参考 `multi_agent_system/src/examples/customer_service_system.py`
基于实际项目的智能客服系统需求分析:
**核心数据结构**:
- `CustomerServicePriority`: 客服优先级管理(低、中、高、紧急)
- `TicketStatus`: 工单状态跟踪(开放、进行中、待处理、已解决、已关闭)
- `CustomerSentiment`: 客户情绪分析(非常负面到非常正面)
- `CustomerProfile`: 客户档案管理(VIP等级、历史记录、满意度评分)
- `SupportTicket`: 支持工单完整生命周期管理
**功能需求**:
- 智能对话:多轮对话,意图识别准确率 > 95%
- 知识检索:向量数据库 + 语义搜索,响应时间 < 500ms
- 工单处理:工作流引擎 + 状态机,处理效率提升 50%
- 情感分析:情感识别准确率 > 90%,智能转人工
- 多渠道接入:支持 5+ 渠道(网页、微信、电话等)
- VIP客户识别:优先级队列,响应时间 < 30秒
**非功能需求**:
- 性能:响应时间 < 2秒,并发用户 > 1000,吞吐量 > 10000 QPS
- 可用性:99.9% 系统可用性,7x24小时服务,故障恢复 < 5分钟
- 安全性:数据加密、RBAC权限管理、完整审计日志
- 可扩展性:自动扩容、插件化架构、多租户支持
# 实际项目架构实现
class ProductionSystemArchitecture:
"""生产级系统架构(基于实际项目)"""
def __init__(self):
self.core_agents = {
"CustomerServiceAgent": {
"file": "customer_service_system.py",
"description": "主要客服智能体",
"capabilities": [
"客户咨询处理",
"工单管理",
"情感分析",
"知识检索",
"VIP客户识别"
]
},
"BaseAgent": {
"file": "base_agent.py",
"description": "BDI智能体基础架构",
"capabilities": [
"信念管理",
"愿望规划",
"意图执行",
"状态监控"
]
}
}
self.infrastructure_components = {
"消息总线": {
"file": "message_bus.py",
"technology": "异步消息队列",
"features": ["发布订阅", "消息持久化", "负载均衡"]
},
"工作流引擎": {
"file": "langgraph_workflow.py",
"technology": "LangGraph",
"features": ["状态管理", "条件分支", "错误恢复"]
},
"监控系统": {
"file": "langsmith_integration.py",
"technology": "LangSmith",
"features": ["链路追踪", "性能监控", "告警系统"]
},
"数据存储": {
"components": ["PostgreSQL", "Redis", "Elasticsearch"],
"features": ["关系数据", "缓存", "全文搜索"]
}
}
self.deployment_architecture = {
"容器化": {
"technology": "Docker + Kubernetes",
"benefits": ["环境一致性", "快速部署", "弹性扩容"]
},
"微服务": {
"pattern": "每个智能体独立服务",
"benefits": ["独立扩展", "故障隔离", "技术多样性"]
},
"云原生": {
"features": ["自动扩容", "服务发现", "配置管理"],
"platforms": ["Kubernetes", "Docker Swarm", "云平台"]
}
}
# 实际部署配置示例
class ProductionDeploymentConfig:
"""生产环境部署配置"""
@staticmethod
def get_docker_compose_config():
"""Docker Compose 配置"""
return """
version: '3.8'
services:
# 主应用服务
multi-agent-system:
build: .
ports:
- "8000:8000"
environment:
- ENVIRONMENT=production
- DATABASE_URL=postgresql://user:pass@postgres:5432/multiagent
- REDIS_URL=redis://redis:6379
- LANGSMITH_API_KEY=${LANGSMITH_API_KEY}
depends_on:
- postgres
- redis
- elasticsearch
volumes:
- ./logs:/app/logs
- ./data:/app/data
restart: unless-stopped
# 数据库服务
postgres:
image: postgres:15
environment:
- POSTGRES_DB=multiagent
- POSTGRES_USER=user
- POSTGRES_PASSWORD=pass
volumes:
- postgres_data:/var/lib/postgresql/data
ports:
- "5432:5432"
restart: unless-stopped
# 缓存服务
redis:
image: redis:7-alpine
ports:
- "6379:6379"
volumes:
- redis_data:/data
restart: unless-stopped
# 搜索引擎
elasticsearch:
image: elasticsearch:8.8.0
environment:
- discovery.type=single-node
- xpack.security.enabled=false
ports:
- "9200:9200"
volumes:
- es_data:/usr/share/elasticsearch/data
restart: unless-stopped
# 监控服务
prometheus:
image: prom/prometheus
ports:
- "9090:9090"
volumes:
- ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml
- prometheus_data:/prometheus
restart: unless-stopped
# 可视化面板
grafana:
image: grafana/grafana
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_PASSWORD=admin
volumes:
- grafana_data:/var/lib/grafana
- ./monitoring/grafana:/etc/grafana/provisioning
restart: unless-stopped
volumes:
postgres_data:
redis_data:
es_data:
prometheus_data:
grafana_data:
"""
@staticmethod
def get_kubernetes_config():
"""Kubernetes 部署配置"""
return """
apiVersion: apps/v1
kind: Deployment
metadata:
name: multi-agent-system
labels:
app: multi-agent-system
spec:
replicas: 3
selector:
matchLabels:
app: multi-agent-system
template:
metadata:
labels:
app: multi-agent-system
spec:
containers:
- name: multi-agent-system
image: multi-agent-system:latest
ports:
- containerPort: 8000
env:
- name: ENVIRONMENT
value: "production"
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: db-secret
key: url
- name: REDIS_URL
value: "redis://redis-service:6379"
- name: LANGSMITH_API_KEY
valueFrom:
secretKeyRef:
name: langsmith-secret
key: api-key
resources:
requests:
memory: "512Mi"
cpu: "250m"
limits:
memory: "1Gi"
cpu: "500m"
livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /ready
port: 8000
initialDelaySeconds: 5
periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
name: multi-agent-system-service
spec:
selector:
app: multi-agent-system
ports:
- protocol: TCP
port: 80
targetPort: 8000
type: LoadBalancer
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: multi-agent-system-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: multi-agent-system
minReplicas: 3
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
"""
1.2 生产环境部署实践¶
1.2.1 容器化部署¶
# 基于实际项目的 Dockerfile
class ProductionDockerfile:
"""生产级 Dockerfile 配置"""
@staticmethod
def get_dockerfile():
return """
# 多阶段构建,优化镜像大小
FROM python:3.11-slim as builder
# 设置工作目录
WORKDIR /app
# 安装系统依赖
RUN apt-get update && apt-get install -y \\
gcc \\
g++ \\
&& rm -rf /var/lib/apt/lists/*
# 复制依赖文件
COPY requirements.txt .
# 安装Python依赖
RUN pip install --no-cache-dir --user -r requirements.txt
# 生产阶段
FROM python:3.11-slim
# 创建非root用户
RUN useradd --create-home --shell /bin/bash app
# 设置工作目录
WORKDIR /app
# 从builder阶段复制依赖
COPY --from=builder /root/.local /home/app/.local
# 复制应用代码
COPY . .
# 设置权限
RUN chown -R app:app /app
# 切换到非root用户
USER app
# 设置环境变量
ENV PATH=/home/app/.local/bin:$PATH
ENV PYTHONPATH=/app
# 健康检查
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \\
CMD python -c "import requests; requests.get('http://localhost:8000/health')"
# 暴露端口
EXPOSE 8000
# 启动命令
CMD ["python", "main.py"]
"""
# 生产环境启动脚本
class ProductionStartupScript:
"""生产环境启动脚本"""
@staticmethod
def get_startup_script():
return """
#!/bin/bash
# 基于项目 multi_agent_system/main.py 的启动流程
set -e
echo "Starting Multi-Agent System in Production Mode..."
# 环境检查
echo "Checking environment..."
python -c "
import sys
import os
from src.agents.base_agent import BaseAgent
from src.examples.customer_service_system import CustomerServiceAgent
from src.communication.message_bus import MessageBus
from src.workflows.langgraph_workflow import StateGraph
from src.monitoring.langsmith_integration import EnterpriseTracing
print('✓ All modules imported successfully')
print(f'✓ Python version: {sys.version}')
print(f'✓ Environment: {os.getenv(\"ENVIRONMENT\", \"development\")}')
"
# 数据库迁移
echo "Running database migrations..."
python -c "
# from src.database.migrations import run_migrations # 示例代码,实际项目中需根据数据库方案调整
run_migrations()
print('✓ Database migrations completed')
"
# 预热系统
echo "Warming up system..."
python -c "
import asyncio
from main import MultiAgentSystem
async def warmup():
system = MultiAgentSystem()
await system.initialize()
print('✓ System warmup completed')
await system.shutdown()
asyncio.run(warmup())
"
# 启动主应用
echo "Starting main application..."
exec python main.py
"""
1.2 核心智能体实现¶
1.2.1 对话管理智能体¶
from langgraph import StateGraph, END
from langchain.schema import BaseMessage
from typing import Dict, List, Any
import asyncio
class DialogManagerAgent:
"""对话管理智能体"""
def __init__(self):
self.conversation_history = {}
self.context_window = 10
async def process_message(self, user_id: str, message: str) -> Dict[str, Any]:
"""处理用户消息"""
# 获取对话历史
history = self.get_conversation_history(user_id)
# 构建状态图
workflow = StateGraph(ConversationState)
# 添加节点
workflow.add_node("intent_recognition", self.recognize_intent)
workflow.add_node("knowledge_retrieval", self.retrieve_knowledge)
workflow.add_node("response_generation", self.generate_response)
workflow.add_node("sentiment_analysis", self.analyze_sentiment)
# 定义边
workflow.add_edge("intent_recognition", "knowledge_retrieval")
workflow.add_edge("knowledge_retrieval", "response_generation")
workflow.add_edge("response_generation", "sentiment_analysis")
workflow.add_edge("sentiment_analysis", END)
# 设置入口点
workflow.set_entry_point("intent_recognition")
# 编译并运行
app = workflow.compile()
initial_state = ConversationState(
user_id=user_id,
message=message,
history=history,
context={}
)
result = await app.ainvoke(initial_state)
return result
class ConversationState:
"""对话状态"""
def __init__(self, user_id: str, message: str, history: List, context: Dict):
self.user_id = user_id
self.message = message
self.history = history
self.context = context
self.intent = None
self.knowledge = None
self.response = None
self.sentiment = None
1.2.2 知识检索智能体¶
from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings
from langchain.text_splitter import RecursiveCharacterTextSplitter
class KnowledgeRetrieverAgent:
"""知识检索智能体"""
def __init__(self):
self.embeddings = OpenAIEmbeddings()
self.vectorstore = None
self.setup_knowledge_base()
def setup_knowledge_base(self):
"""设置知识库"""
# 加载企业知识文档
documents = self.load_knowledge_documents()
# 文档分割
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200
)
splits = text_splitter.split_documents(documents)
# 创建向量存储
self.vectorstore = Chroma.from_documents(
documents=splits,
embedding=self.embeddings,
persist_directory="./knowledge_db"
)
async def retrieve_knowledge(self, query: str, top_k: int = 5) -> List[Dict]:
"""检索相关知识"""
# 相似性搜索
docs = self.vectorstore.similarity_search_with_score(query, k=top_k)
# 重排序
reranked_docs = await self.rerank_documents(query, docs)
return [
{
"content": doc.page_content,
"metadata": doc.metadata,
"score": score
}
for doc, score in reranked_docs
]
async def rerank_documents(self, query: str, docs: List) -> List:
"""文档重排序"""
# 使用更精确的重排序模型
# 这里可以集成 Cohere Rerank 或其他重排序服务
return docs # 简化实现
1.3 系统集成与测试¶
1.3.1 集成测试框架¶
import pytest
import asyncio
from unittest.mock import Mock, patch
class IntegrationTestSuite:
"""集成测试套件"""
@pytest.fixture
async def customer_service_system(self):
"""客服系统测试夹具"""
system = CustomerServiceSystem()
await system.initialize()
yield system
await system.cleanup()
@pytest.mark.asyncio
async def test_end_to_end_conversation(self, customer_service_system):
"""端到端对话测试"""
user_id = "test_user_001"
# 测试场景1:产品咨询
response1 = await customer_service_system.process_message(
user_id, "我想了解你们的产品价格"
)
assert response1["intent"] == "product_inquiry"
assert "价格" in response1["response"]
# 测试场景2:技术支持
response2 = await customer_service_system.process_message(
user_id, "我的账号登录不了"
)
assert response2["intent"] == "technical_support"
assert response2["escalate_to_human"] == False
# 测试场景3:投诉处理
response3 = await customer_service_system.process_message(
user_id, "你们的服务太差了,我要投诉"
)
assert response3["sentiment"] == "negative"
assert response3["escalate_to_human"] == True
@pytest.mark.asyncio
async def test_performance_under_load(self, customer_service_system):
"""负载测试"""
# 模拟100个并发用户
tasks = []
for i in range(100):
task = customer_service_system.process_message(
f"user_{i}", "Hello, I need help"
)
tasks.append(task)
# 执行并发测试
start_time = asyncio.get_event_loop().time()
results = await asyncio.gather(*tasks)
end_time = asyncio.get_event_loop().time()
# 验证性能指标
avg_response_time = (end_time - start_time) / len(results)
assert avg_response_time < 2.0 # 平均响应时间小于2秒
assert all(result["status"] == "success" for result in results)
2. 生产环境部署¶
2.1 容器化部署¶
2.1.1 Docker配置¶
# Dockerfile
FROM python:3.11-slim
WORKDIR /app
# 安装系统依赖
RUN apt-get update && apt-get install -y \
gcc \
g++ \
&& rm -rf /var/lib/apt/lists/*
# 复制依赖文件
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# 复制应用代码
COPY . .
# 设置环境变量
ENV PYTHONPATH=/app
ENV LANGCHAIN_TRACING_V2=true
ENV LANGCHAIN_ENDPOINT=https://api.smith.langchain.com
# 暴露端口
EXPOSE 8000
# 健康检查
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
CMD curl -f http://localhost:8000/health || exit 1
# 启动命令
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
2.1.2 Docker Compose配置¶
# docker-compose.yml
version: '3.8'
services:
customer-service:
build: .
ports:
- "8000:8000"
environment:
- DATABASE_URL=postgresql://user:password@postgres:5432/customer_service
- REDIS_URL=redis://redis:6379
- ELASTICSEARCH_URL=http://elasticsearch:9200
depends_on:
- postgres
- redis
- elasticsearch
volumes:
- ./logs:/app/logs
restart: unless-stopped
postgres:
image: postgres:15
environment:
POSTGRES_DB: customer_service
POSTGRES_USER: user
POSTGRES_PASSWORD: password
volumes:
- postgres_data:/var/lib/postgresql/data
ports:
- "5432:5432"
redis:
image: redis:7-alpine
ports:
- "6379:6379"
volumes:
- redis_data:/data
elasticsearch:
image: elasticsearch:8.8.0
environment:
- discovery.type=single-node
- xpack.security.enabled=false
ports:
- "9200:9200"
volumes:
- elasticsearch_data:/usr/share/elasticsearch/data
nginx:
image: nginx:alpine
ports:
- "80:80"
- "443:443"
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf
- ./ssl:/etc/nginx/ssl
depends_on:
- customer-service
volumes:
postgres_data:
redis_data:
elasticsearch_data:
2.2 Kubernetes部署¶
2.2.1 K8s部署配置¶
# k8s-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: customer-service
labels:
app: customer-service
spec:
replicas: 3
selector:
matchLabels:
app: customer-service
template:
metadata:
labels:
app: customer-service
spec:
containers:
- name: customer-service
image: customer-service:latest
ports:
- containerPort: 8000
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: db-secret
key: url
- name: REDIS_URL
value: "redis://redis-service:6379"
resources:
requests:
memory: "512Mi"
cpu: "250m"
limits:
memory: "1Gi"
cpu: "500m"
livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /ready
port: 8000
initialDelaySeconds: 5
periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
name: customer-service
spec:
selector:
app: customer-service
ports:
- port: 80
targetPort: 8000
type: LoadBalancer
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: customer-service-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: customer-service
minReplicas: 3
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
3. 企业级监控与运维¶
3.1 基于实际项目的监控系统¶
3.1.1 企业级监控架构¶
# 基于项目 multi_agent_system/src/monitoring/langsmith_integration.py 的监控实现
import asyncio
import time
import psutil
import logging
from typing import Dict, List, Any, Optional
from dataclasses import dataclass, asdict
from datetime import datetime, timedelta
from prometheus_client import Counter, Histogram, Gauge, start_http_server
@dataclass
class SystemMetrics:
"""系统指标"""
timestamp: datetime
cpu_usage: float
memory_usage: float
disk_usage: float
network_io: Dict[str, int]
active_connections: int
response_time_p95: float
error_rate: float
throughput: float
@dataclass
class AgentMetrics:
"""智能体指标"""
agent_id: str
agent_type: str
total_requests: int
successful_requests: int
failed_requests: int
avg_response_time: float
current_load: float
memory_usage: float
last_activity: datetime
class EnterpriseMonitoringSystem:
"""企业级监控系统(基于实际项目)"""
def __init__(self):
self.setup_prometheus_metrics()
self.setup_logging()
self.alert_manager = AlertManager()
self.performance_analyzer = PerformanceAnalyzer()
def setup_prometheus_metrics(self):
"""设置 Prometheus 指标"""
# 请求指标
self.request_counter = Counter(
'multiagent_requests_total',
'Total number of requests',
['agent_type', 'status']
)
self.request_duration = Histogram(
'multiagent_request_duration_seconds',
'Request duration in seconds',
['agent_type', 'endpoint']
)
# 系统指标
self.cpu_usage = Gauge(
'multiagent_cpu_usage_percent',
'CPU usage percentage'
)
self.memory_usage = Gauge(
'multiagent_memory_usage_bytes',
'Memory usage in bytes'
)
self.active_agents = Gauge(
'multiagent_active_agents',
'Number of active agents',
['agent_type']
)
# 业务指标
self.customer_satisfaction = Gauge(
'multiagent_customer_satisfaction_score',
'Customer satisfaction score'
)
self.ticket_resolution_time = Histogram(
'multiagent_ticket_resolution_seconds',
'Ticket resolution time in seconds',
['priority', 'category']
)
def setup_logging(self):
"""设置日志系统"""
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
handlers=[
logging.FileHandler('/app/logs/multiagent.log'),
logging.StreamHandler()
]
)
self.logger = logging.getLogger(__name__)
async def collect_system_metrics(self) -> SystemMetrics:
"""收集系统指标"""
# CPU 使用率
cpu_usage = psutil.cpu_percent(interval=1)
# 内存使用率
memory = psutil.virtual_memory()
memory_usage = memory.percent
# 磁盘使用率
disk = psutil.disk_usage('/')
disk_usage = disk.percent
# 网络IO
network = psutil.net_io_counters()
network_io = {
'bytes_sent': network.bytes_sent,
'bytes_recv': network.bytes_recv
}
# 活跃连接数
connections = len(psutil.net_connections())
# 更新 Prometheus 指标
self.cpu_usage.set(cpu_usage)
self.memory_usage.set(memory.used)
return SystemMetrics(
timestamp=datetime.now(),
cpu_usage=cpu_usage,
memory_usage=memory_usage,
disk_usage=disk_usage,
network_io=network_io,
active_connections=connections,
response_time_p95=0.0, # 从应用指标获取
error_rate=0.0, # 从应用指标获取
throughput=0.0 # 从应用指标获取
)
async def monitor_agent_performance(self, agent_id: str) -> AgentMetrics:
"""监控智能体性能"""
# 从实际项目的智能体获取指标
agent_stats = await self.get_agent_statistics(agent_id)
metrics = AgentMetrics(
agent_id=agent_id,
agent_type=agent_stats.get('type', 'unknown'),
total_requests=agent_stats.get('total_requests', 0),
successful_requests=agent_stats.get('successful_requests', 0),
failed_requests=agent_stats.get('failed_requests', 0),
avg_response_time=agent_stats.get('avg_response_time', 0.0),
current_load=agent_stats.get('current_load', 0.0),
memory_usage=agent_stats.get('memory_usage', 0.0),
last_activity=datetime.now()
)
# 更新 Prometheus 指标
self.active_agents.labels(agent_type=metrics.agent_type).set(1)
return metrics
class AlertManager:
"""告警管理器"""
def __init__(self):
self.alert_rules = self.load_alert_rules()
self.notification_channels = self.setup_notification_channels()
def load_alert_rules(self) -> List[Dict]:
"""加载告警规则"""
return [
{
"name": "high_cpu_usage",
"condition": "cpu_usage > 80",
"severity": "warning",
"duration": "5m",
"message": "CPU usage is above 80% for 5 minutes"
},
{
"name": "high_memory_usage",
"condition": "memory_usage > 90",
"severity": "critical",
"duration": "2m",
"message": "Memory usage is above 90% for 2 minutes"
},
{
"name": "high_error_rate",
"condition": "error_rate > 5",
"severity": "critical",
"duration": "1m",
"message": "Error rate is above 5% for 1 minute"
},
{
"name": "slow_response_time",
"condition": "response_time_p95 > 5",
"severity": "warning",
"duration": "3m",
"message": "95th percentile response time is above 5 seconds"
},
{
"name": "agent_down",
"condition": "active_agents == 0",
"severity": "critical",
"duration": "30s",
"message": "No active agents detected"
}
]
class PerformanceAnalyzer:
"""性能分析器"""
def __init__(self):
self.metrics_history = []
self.analysis_window = timedelta(hours=1)
async def analyze_performance_trends(self, metrics: SystemMetrics) -> Dict[str, Any]:
"""分析性能趋势"""
self.metrics_history.append(metrics)
# 保持分析窗口大小
cutoff_time = datetime.now() - self.analysis_window
self.metrics_history = [
m for m in self.metrics_history
if m.timestamp > cutoff_time
]
if len(self.metrics_history) < 10:
return {"status": "insufficient_data"}
# 计算趋势
cpu_trend = self.calculate_trend([m.cpu_usage for m in self.metrics_history])
memory_trend = self.calculate_trend([m.memory_usage for m in self.metrics_history])
response_time_trend = self.calculate_trend([m.response_time_p95 for m in self.metrics_history])
# 异常检测
anomalies = self.detect_anomalies()
# 性能建议
recommendations = self.generate_recommendations(cpu_trend, memory_trend, response_time_trend)
return {
"status": "analysis_complete",
"trends": {
"cpu": cpu_trend,
"memory": memory_trend,
"response_time": response_time_trend
},
"anomalies": anomalies,
"recommendations": recommendations,
"analysis_period": {
"start": self.metrics_history[0].timestamp.isoformat(),
"end": self.metrics_history[-1].timestamp.isoformat(),
"sample_count": len(self.metrics_history)
}
}
# 生产环境监控配置
class ProductionMonitoringConfig:
"""生产环境监控配置"""
@staticmethod
def get_prometheus_config():
"""Prometheus 配置"""
return """
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
- "alert_rules.yml"
alerting:
alertmanagers:
- static_configs:
- targets:
- alertmanager:9093
scrape_configs:
- job_name: 'multi-agent-system'
static_configs:
- targets: ['multi-agent-system:8000']
metrics_path: '/metrics'
scrape_interval: 10s
- job_name: 'node-exporter'
static_configs:
- targets: ['node-exporter:9100']
- job_name: 'postgres-exporter'
static_configs:
- targets: ['postgres-exporter:9187']
- job_name: 'redis-exporter'
static_configs:
- targets: ['redis-exporter:9121']
"""
@staticmethod
def get_grafana_dashboard():
"""Grafana 仪表板配置"""
return {
"dashboard": {
"title": "Multi-Agent System Monitoring",
"panels": [
{
"title": "System Overview",
"type": "stat",
"targets": [
{"expr": "multiagent_cpu_usage_percent"},
{"expr": "multiagent_memory_usage_bytes / 1024 / 1024 / 1024"},
{"expr": "rate(multiagent_requests_total[5m])"}
]
},
{
"title": "Request Rate",
"type": "graph",
"targets": [
{"expr": "rate(multiagent_requests_total[5m])"}
]
},
{
"title": "Response Time",
"type": "graph",
"targets": [
{"expr": "histogram_quantile(0.95, multiagent_request_duration_seconds_bucket)"},
{"expr": "histogram_quantile(0.50, multiagent_request_duration_seconds_bucket)"}
]
},
{
"title": "Error Rate",
"type": "graph",
"targets": [
{"expr": "rate(multiagent_requests_total{status=\"error\"}[5m]) / rate(multiagent_requests_total[5m]) * 100"}
]
},
{
"title": "Active Agents",
"type": "stat",
"targets": [
{"expr": "multiagent_active_agents"}
]
},
{
"title": "Customer Satisfaction",
"type": "gauge",
"targets": [
{"expr": "multiagent_customer_satisfaction_score"}
]
}
]
}
}
# 启动监控系统
async def start_monitoring_system():
"""启动监控系统"""
monitoring = EnterpriseMonitoringSystem()
# 启动 Prometheus 指标服务器
start_http_server(8001)
# 监控循环
while True:
try:
# 收集系统指标
system_metrics = await monitoring.collect_system_metrics()
# 检查告警
await monitoring.alert_manager.check_alerts(system_metrics)
# 性能分析
analysis = await monitoring.performance_analyzer.analyze_performance_trends(system_metrics)
# 记录日志
monitoring.logger.info(f"System metrics collected: {system_metrics}")
if analysis.get("status") == "analysis_complete":
monitoring.logger.info(f"Performance analysis: {analysis}")
# 等待下一次收集
await asyncio.sleep(30)
except Exception as e:
monitoring.logger.error(f"Monitoring error: {e}")
await asyncio.sleep(60)
3.2 自动化运维实践¶
3.2.1 自动扩缩容¶
# 基于实际项目的自动扩缩容实现
import kubernetes
from kubernetes import client, config
import asyncio
import logging
from typing import Dict, List, Any
from dataclasses import dataclass
from datetime import datetime, timedelta
@dataclass
class ScalingMetrics:
"""扩缩容指标"""
cpu_utilization: float
memory_utilization: float
request_rate: float
response_time_p95: float
error_rate: float
queue_length: int
active_connections: int
@dataclass
class ScalingDecision:
"""扩缩容决策"""
action: str # "scale_up", "scale_down", "no_action"
target_replicas: int
reason: str
confidence: float
timestamp: datetime
class AutoScaler:
"""自动扩缩容器"""
def __init__(self):
self.setup_kubernetes_client()
self.scaling_config = self.load_scaling_config()
self.metrics_history = []
self.logger = logging.getLogger(__name__)
def setup_kubernetes_client(self):
"""设置 Kubernetes 客户端"""
try:
# 尝试集群内配置
config.load_incluster_config()
except:
# 使用本地配置
config.load_kube_config()
self.apps_v1 = client.AppsV1Api()
self.core_v1 = client.CoreV1Api()
def load_scaling_config(self) -> Dict[str, Any]:
"""加载扩缩容配置"""
return {
"deployment_name": "multi-agent-system",
"namespace": "default",
"min_replicas": 2,
"max_replicas": 20,
"target_cpu_utilization": 70,
"target_memory_utilization": 80,
"target_response_time": 2.0,
"scale_up_threshold": {
"cpu": 80,
"memory": 85,
"response_time": 3.0,
"error_rate": 5.0
},
"scale_down_threshold": {
"cpu": 30,
"memory": 40,
"response_time": 1.0,
"error_rate": 1.0
},
"cooldown_period": 300, # 5分钟冷却期
"scale_up_factor": 1.5,
"scale_down_factor": 0.8
}
async def make_scaling_decision(self, metrics: ScalingMetrics) -> ScalingDecision:
"""制定扩缩容决策"""
current_replicas = await self.get_current_replicas()
# 检查冷却期
if not self.is_cooldown_expired():
return ScalingDecision(
action="no_action",
target_replicas=current_replicas,
reason="在冷却期内",
confidence=1.0,
timestamp=datetime.now()
)
# 扩容条件检查
scale_up_score = self.calculate_scale_up_score(metrics)
scale_down_score = self.calculate_scale_down_score(metrics)
if scale_up_score > 0.7:
target_replicas = min(
int(current_replicas * self.scaling_config["scale_up_factor"]),
self.scaling_config["max_replicas"]
)
return ScalingDecision(
action="scale_up",
target_replicas=target_replicas,
reason=f"扩容评分: {scale_up_score:.2f}",
confidence=scale_up_score,
timestamp=datetime.now()
)
elif scale_down_score > 0.7:
target_replicas = max(
int(current_replicas * self.scaling_config["scale_down_factor"]),
self.scaling_config["min_replicas"]
)
return ScalingDecision(
action="scale_down",
target_replicas=target_replicas,
reason=f"缩容评分: {scale_down_score:.2f}",
confidence=scale_down_score,
timestamp=datetime.now()
)
return ScalingDecision(
action="no_action",
target_replicas=current_replicas,
reason="指标正常",
confidence=0.5,
timestamp=datetime.now()
)
# 自动化运维主循环
async def auto_scaling_loop():
"""自动扩缩容主循环"""
autoscaler = AutoScaler()
while True:
try:
# 收集指标
metrics = await autoscaler.collect_scaling_metrics()
# 制定决策
decision = await autoscaler.make_scaling_decision(metrics)
# 执行扩缩容
success = await autoscaler.execute_scaling(decision)
if success:
autoscaler.logger.info(f"扩缩容决策: {decision}")
# 等待下一次检查
await asyncio.sleep(60) # 每分钟检查一次
except Exception as e:
autoscaler.logger.error(f"自动扩缩容错误: {e}")
await asyncio.sleep(300) # 错误时等待5分钟
3.3 故障恢复与灾备¶
3.3.1 故障检测与自动恢复¶
# 基于实际项目的故障恢复机制
import asyncio
import logging
from typing import Dict, List, Any, Optional
from dataclasses import dataclass
from datetime import datetime, timedelta
from enum import Enum
class FailureType(Enum):
"""故障类型"""
SERVICE_DOWN = "service_down"
HIGH_ERROR_RATE = "high_error_rate"
SLOW_RESPONSE = "slow_response"
MEMORY_LEAK = "memory_leak"
DATABASE_CONNECTION = "database_connection"
NETWORK_ISSUE = "network_issue"
class RecoveryAction(Enum):
"""恢复动作"""
RESTART_SERVICE = "restart_service"
SCALE_UP = "scale_up"
FAILOVER = "failover"
CIRCUIT_BREAKER = "circuit_breaker"
ROLLBACK = "rollback"
MANUAL_INTERVENTION = "manual_intervention"
@dataclass
class FailureEvent:
"""故障事件"""
failure_type: FailureType
severity: str
description: str
affected_services: List[str]
timestamp: datetime
metrics: Dict[str, Any]
recovery_actions: List[RecoveryAction]
class FailureDetector:
"""故障检测器"""
def __init__(self):
self.detection_rules = self.load_detection_rules()
self.logger = logging.getLogger(__name__)
def load_detection_rules(self) -> Dict[FailureType, Dict]:
"""加载故障检测规则"""
return {
FailureType.SERVICE_DOWN: {
"condition": "health_check_failures > 3",
"window": "2m",
"severity": "critical"
},
FailureType.HIGH_ERROR_RATE: {
"condition": "error_rate > 10",
"window": "5m",
"severity": "high"
},
FailureType.SLOW_RESPONSE: {
"condition": "response_time_p95 > 10",
"window": "5m",
"severity": "medium"
},
FailureType.MEMORY_LEAK: {
"condition": "memory_growth_rate > 5",
"window": "30m",
"severity": "high"
}
}
async def detect_failures(self, metrics: Dict[str, Any]) -> List[FailureEvent]:
"""检测故障"""
failures = []
for failure_type, rule in self.detection_rules.items():
if self.evaluate_condition(rule["condition"], metrics):
failure = FailureEvent(
failure_type=failure_type,
severity=rule["severity"],
description=f"检测到{failure_type.value}故障",
affected_services=self.identify_affected_services(failure_type, metrics),
timestamp=datetime.now(),
metrics=metrics,
recovery_actions=self.determine_recovery_actions(failure_type)
)
failures.append(failure)
return failures
def determine_recovery_actions(self, failure_type: FailureType) -> List[RecoveryAction]:
"""确定恢复动作"""
recovery_map = {
FailureType.SERVICE_DOWN: [RecoveryAction.RESTART_SERVICE, RecoveryAction.FAILOVER],
FailureType.HIGH_ERROR_RATE: [RecoveryAction.CIRCUIT_BREAKER, RecoveryAction.ROLLBACK],
FailureType.SLOW_RESPONSE: [RecoveryAction.SCALE_UP, RecoveryAction.CIRCUIT_BREAKER],
FailureType.MEMORY_LEAK: [RecoveryAction.RESTART_SERVICE],
FailureType.DATABASE_CONNECTION: [RecoveryAction.FAILOVER, RecoveryAction.RESTART_SERVICE],
FailureType.NETWORK_ISSUE: [RecoveryAction.FAILOVER, RecoveryAction.MANUAL_INTERVENTION]
}
return recovery_map.get(failure_type, [RecoveryAction.MANUAL_INTERVENTION])
class AutoRecoverySystem:
"""自动恢复系统"""
def __init__(self):
self.failure_detector = FailureDetector()
self.recovery_executor = RecoveryExecutor()
self.logger = logging.getLogger(__name__)
async def monitor_and_recover(self):
"""监控和恢复主循环"""
while True:
try:
# 收集系统指标
metrics = await self.collect_system_metrics()
# 检测故障
failures = await self.failure_detector.detect_failures(metrics)
# 执行恢复动作
for failure in failures:
await self.handle_failure(failure)
await asyncio.sleep(30)
except Exception as e:
self.logger.error(f"监控恢复系统错误: {e}")
await asyncio.sleep(60)
async def handle_failure(self, failure: FailureEvent):
"""处理故障"""
self.logger.warning(f"检测到故障: {failure}")
for action in failure.recovery_actions:
try:
success = await self.recovery_executor.execute_action(action, failure)
if success:
self.logger.info(f"恢复动作 {action} 执行成功")
break
else:
self.logger.warning(f"恢复动作 {action} 执行失败")
except Exception as e:
self.logger.error(f"执行恢复动作 {action} 时出错: {e}")
# 发送告警通知
await self.send_failure_notification(failure)
class RecoveryExecutor:
"""恢复执行器"""
async def execute_action(self, action: RecoveryAction, failure: FailureEvent) -> bool:
"""执行恢复动作"""
if action == RecoveryAction.RESTART_SERVICE:
return await self.restart_service(failure.affected_services)
elif action == RecoveryAction.SCALE_UP:
return await self.scale_up_service(failure.affected_services)
elif action == RecoveryAction.FAILOVER:
return await self.failover_service(failure.affected_services)
elif action == RecoveryAction.CIRCUIT_BREAKER:
return await self.enable_circuit_breaker(failure.affected_services)
elif action == RecoveryAction.ROLLBACK:
return await self.rollback_deployment(failure.affected_services)
else:
return False
async def restart_service(self, services: List[str]) -> bool:
"""重启服务"""
try:
for service in services:
# 使用 Kubernetes API 重启服务
await self.kubernetes_restart_deployment(service)
return True
except Exception as e:
logging.error(f"重启服务失败: {e}")
return False
async def kubernetes_restart_deployment(self, deployment_name: str):
"""Kubernetes 重启部署"""
# 实际的 Kubernetes API 调用
pass
3.4 性能优化实践¶
3.4.1 系统性能调优¶
# 基于实际项目的性能优化实现
import asyncio
import cProfile
import pstats
import io
import logging
from typing import Dict, List, Any, Optional
from dataclasses import dataclass
from datetime import datetime
import psutil
import gc
from prometheus_client import Counter, Gauge, Histogram
# Prometheus 指标定义
REQUEST_COUNT = Counter('requests_total', 'Total requests', ['method', 'endpoint'])
ACTIVE_CONVERSATIONS = Gauge('active_conversations', 'Number of active conversations')
RESPONSE_TIME = Histogram('response_time_seconds', 'Response time in seconds')
def monitor_performance(operation_name: str):
"""性能监控装饰器"""
def decorator(func):
async def wrapper(*args, **kwargs):
start_time = datetime.now()
try:
result = await func(*args, **kwargs)
return result
finally:
duration = (datetime.now() - start_time).total_seconds()
RESPONSE_TIME.observe(duration)
logging.info(f"{operation_name} 执行时间: {duration:.2f}秒")
return wrapper
return decorator
@dataclass
class PerformanceProfile:
"""性能分析结果"""
function_stats: Dict[str, Any]
memory_usage: Dict[str, float]
cpu_usage: float
io_stats: Dict[str, int]
bottlenecks: List[str]
recommendations: List[str]
class PerformanceOptimizer:
"""性能优化器"""
def __init__(self):
self.profiler = cProfile.Profile()
self.optimization_history = []
self.logger = logging.getLogger(__name__)
async def profile_system_performance(self) -> PerformanceProfile:
"""分析系统性能"""
# 开始性能分析
self.profiler.enable()
# 运行一段时间收集数据
await asyncio.sleep(60)
# 停止分析
self.profiler.disable()
# 分析结果
s = io.StringIO()
ps = pstats.Stats(self.profiler, stream=s)
ps.sort_stats('cumulative')
ps.print_stats()
# 收集系统指标
memory_info = psutil.virtual_memory()
cpu_usage = psutil.cpu_percent()
io_counters = psutil.disk_io_counters()
# 识别瓶颈
bottlenecks = self.identify_bottlenecks(ps)
# 生成优化建议
recommendations = self.generate_optimization_recommendations(bottlenecks)
return PerformanceProfile(
function_stats=self.extract_function_stats(ps),
memory_usage={
"total": memory_info.total,
"used": memory_info.used,
"percent": memory_info.percent
},
cpu_usage=cpu_usage,
io_stats={
"read_bytes": io_counters.read_bytes,
"write_bytes": io_counters.write_bytes
},
bottlenecks=bottlenecks,
recommendations=recommendations
)
def identify_bottlenecks(self, stats: pstats.Stats) -> List[str]:
"""识别性能瓶颈"""
bottlenecks = []
# 分析最耗时的函数
stats.sort_stats('cumulative')
top_functions = stats.get_stats_profile().func_profiles
for func, (cc, nc, tt, ct, callers) in list(top_functions.items())[:10]:
if ct > 1.0: # 累计时间超过1秒
bottlenecks.append(f"函数 {func} 耗时过长: {ct:.2f}秒")
return bottlenecks
def generate_optimization_recommendations(self, bottlenecks: List[str]) -> List[str]:
"""生成优化建议"""
recommendations = []
for bottleneck in bottlenecks:
if "数据库" in bottleneck:
recommendations.append("优化数据库查询,添加索引或使用缓存")
elif "网络" in bottleneck:
recommendations.append("优化网络请求,使用连接池或异步处理")
elif "内存" in bottleneck:
recommendations.append("优化内存使用,检查内存泄漏")
elif "CPU" in bottleneck:
recommendations.append("优化算法复杂度,使用并行处理")
else:
recommendations.append("进一步分析具体瓶颈原因")
return recommendations
async def apply_optimizations(self, profile: PerformanceProfile):
"""应用性能优化"""
for recommendation in profile.recommendations:
if "缓存" in recommendation:
await self.optimize_caching()
elif "数据库" in recommendation:
await self.optimize_database()
elif "内存" in recommendation:
await self.optimize_memory()
elif "并行" in recommendation:
await self.optimize_concurrency()
async def optimize_caching(self):
"""优化缓存策略"""
# 实现缓存优化逻辑
self.logger.info("应用缓存优化策略")
async def optimize_database(self):
"""优化数据库性能"""
# 实现数据库优化逻辑
self.logger.info("应用数据库优化策略")
async def optimize_memory(self):
"""优化内存使用"""
# 强制垃圾回收
gc.collect()
self.logger.info("执行内存优化")
async def optimize_concurrency(self):
"""优化并发处理"""
# 实现并发优化逻辑
self.logger.info("应用并发优化策略")
def extract_function_stats(self, stats: pstats.Stats) -> Dict[str, Any]:
"""提取函数统计信息"""
function_stats = {}
stats_profile = stats.get_stats_profile()
for func, (cc, nc, tt, ct, callers) in stats_profile.func_profiles.items():
function_stats[str(func)] = {
"call_count": cc,
"total_time": tt,
"cumulative_time": ct,
"per_call_time": tt / cc if cc > 0 else 0
}
return function_stats
# 性能监控和优化主循环
async def performance_optimization_loop():
"""性能优化主循环"""
optimizer = PerformanceOptimizer()
while True:
try:
# 分析性能
profile = await optimizer.profile_system_performance()
# 记录分析结果
optimizer.logger.info(f"性能分析完成: {profile}")
# 应用优化
await optimizer.apply_optimizations(profile)
# 等待下一次分析
await asyncio.sleep(3600) # 每小时分析一次
except Exception as e:
optimizer.logger.error(f"性能优化错误: {e}")
await asyncio.sleep(1800) # 错误时等待30分钟
# 使用示例
async def run_performance_optimization():
"""运行性能优化示例"""
optimizer = PerformanceOptimizer()
# 启动性能优化循环
await performance_optimization_loop()
# 带监控的对话管理器示例
class MonitoredDialogManager:
"""带监控的对话管理器"""
def __init__(self):
self.logger = logging.getLogger(__name__)
@monitor_performance("dialog_manager")
async def process_message(self, user_id: str, message: str):
"""处理用户消息"""
ACTIVE_CONVERSATIONS.inc()
try:
# 模拟消息处理逻辑
result = await self._handle_message(user_id, message)
REQUEST_COUNT.labels(method='POST', endpoint='/chat').inc()
return result
finally:
ACTIVE_CONVERSATIONS.dec()
async def _handle_message(self, user_id: str, message: str) -> str:
"""处理消息的具体逻辑"""
# 这里实现具体的消息处理逻辑
await asyncio.sleep(0.1) # 模拟处理时间
return f"处理用户 {user_id} 的消息: {message}"
3.2 日志管理¶
3.2.1 结构化日志配置¶
import logging
import json
from datetime import datetime
from typing import Dict, Any
class StructuredLogger:
"""结构化日志记录器"""
def __init__(self, name: str):
self.logger = logging.getLogger(name)
self.logger.setLevel(logging.INFO)
# 配置处理器
handler = logging.StreamHandler()
formatter = logging.Formatter('%(message)s')
handler.setFormatter(formatter)
self.logger.addHandler(handler)
def log_event(self, event_type: str, data: Dict[str, Any], level: str = "INFO"):
"""记录结构化事件"""
log_entry = {
"timestamp": datetime.utcnow().isoformat(),
"event_type": event_type,
"level": level,
"data": data
}
if level == "ERROR":
self.logger.error(json.dumps(log_entry))
elif level == "WARNING":
self.logger.warning(json.dumps(log_entry))
else:
self.logger.info(json.dumps(log_entry))
# 使用示例
logger = StructuredLogger("customer_service")
# 记录对话事件
logger.log_event("conversation_started", {
"user_id": "user_123",
"session_id": "session_456",
"channel": "web"
})
# 记录错误事件
logger.log_event("agent_error", {
"agent_type": "knowledge_retriever",
"error_message": "Connection timeout",
"user_id": "user_123"
}, level="ERROR")
3.3 性能优化¶
3.3.1 缓存策略¶
import redis
import pickle
from typing import Optional, Any
import hashlib
class CacheManager:
"""缓存管理器"""
def __init__(self, redis_url: str):
self.redis_client = redis.from_url(redis_url)
self.default_ttl = 3600 # 1小时
def _generate_key(self, prefix: str, *args) -> str:
"""生成缓存键"""
key_data = f"{prefix}:{':'.join(map(str, args))}"
return hashlib.md5(key_data.encode()).hexdigest()
async def get(self, prefix: str, *args) -> Optional[Any]:
"""获取缓存"""
key = self._generate_key(prefix, *args)
data = self.redis_client.get(key)
if data:
return pickle.loads(data)
return None
async def set(self, prefix: str, value: Any, ttl: Optional[int] = None, *args):
"""设置缓存"""
key = self._generate_key(prefix, *args)
ttl = ttl or self.default_ttl
self.redis_client.setex(key, ttl, pickle.dumps(value))
def cache_result(self, prefix: str, ttl: Optional[int] = None):
"""缓存装饰器"""
def decorator(func):
@functools.wraps(func)
async def wrapper(*args, **kwargs):
# 生成缓存键
cache_key = self._generate_key(prefix, *args, **kwargs)
# 尝试从缓存获取
cached_result = await self.get(prefix, *args, **kwargs)
if cached_result is not None:
return cached_result
# 执行函数并缓存结果
result = await func(*args, **kwargs)
await self.set(prefix, result, ttl, *args, **kwargs)
return result
return wrapper
return decorator
# 应用缓存
cache_manager = CacheManager("redis://localhost:6379")
class CachedKnowledgeRetriever(KnowledgeRetrieverAgent):
"""带缓存的知识检索器"""
@cache_manager.cache_result("knowledge_search", ttl=1800)
async def retrieve_knowledge(self, query: str, top_k: int = 5):
return await super().retrieve_knowledge(query, top_k)
4. 容器化部署与云原生运维¶
4.1 完整容器化部署方案¶
4.1.1 多阶段Dockerfile优化¶
# 基于实际项目的多阶段构建Dockerfile
FROM python:3.11-slim as builder
# 设置工作目录
WORKDIR /app
# 安装系统依赖
RUN apt-get update && apt-get install -y \
gcc \
g++ \
&& rm -rf /var/lib/apt/lists/*
# 复制依赖文件
COPY requirements.txt .
# 安装Python依赖
RUN pip install --no-cache-dir --user -r requirements.txt
# 生产阶段
FROM python:3.11-slim as production
# 创建非root用户
RUN groupadd -r appuser && useradd -r -g appuser appuser
# 设置工作目录
WORKDIR /app
# 从builder阶段复制依赖
COPY --from=builder /root/.local /home/appuser/.local
# 复制应用代码
COPY --chown=appuser:appuser . .
# 设置环境变量
ENV PATH=/home/appuser/.local/bin:$PATH
ENV PYTHONPATH=/app
ENV PYTHONUNBUFFERED=1
# 切换到非root用户
USER appuser
# 健康检查
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
CMD python -c "import requests; requests.get('http://localhost:8000/health')"
# 暴露端口
EXPOSE 8000
# 启动命令
CMD ["python", "main.py"]
4.1.2 生产级Docker Compose配置¶
# docker-compose.prod.yml - 生产环境配置
version: '3.8'
services:
# 主应用服务
customer-service:
build:
context: .
dockerfile: Dockerfile
target: production
image: customer-service:latest
container_name: customer-service-app
restart: unless-stopped
environment:
- ENVIRONMENT=production
- REDIS_URL=redis://redis:6379
- DATABASE_URL=postgresql://user:password@postgres:5432/customer_service
- LANGSMITH_API_KEY=${LANGSMITH_API_KEY}
- OPENAI_API_KEY=${OPENAI_API_KEY}
ports:
- "8000:8000"
depends_on:
- redis
- postgres
- prometheus
networks:
- app-network
volumes:
- ./logs:/app/logs
- ./config:/app/config:ro
deploy:
resources:
limits:
cpus: '2.0'
memory: 4G
reservations:
cpus: '1.0'
memory: 2G
# Redis缓存服务
redis:
image: redis:7-alpine
container_name: customer-service-redis
restart: unless-stopped
command: redis-server --appendonly yes --requirepass ${REDIS_PASSWORD}
ports:
- "6379:6379"
volumes:
- redis-data:/data
networks:
- app-network
deploy:
resources:
limits:
memory: 1G
# PostgreSQL数据库
postgres:
image: postgres:15-alpine
container_name: customer-service-db
restart: unless-stopped
environment:
- POSTGRES_DB=customer_service
- POSTGRES_USER=user
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
ports:
- "5432:5432"
volumes:
- postgres-data:/var/lib/postgresql/data
- ./init.sql:/docker-entrypoint-initdb.d/init.sql:ro
networks:
- app-network
deploy:
resources:
limits:
memory: 2G
# Nginx负载均衡器
nginx:
image: nginx:alpine
container_name: customer-service-nginx
restart: unless-stopped
ports:
- "80:80"
- "443:443"
volumes:
- ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
- ./nginx/ssl:/etc/nginx/ssl:ro
- ./logs/nginx:/var/log/nginx
depends_on:
- customer-service
networks:
- app-network
# Prometheus监控
prometheus:
image: prom/prometheus:latest
container_name: customer-service-prometheus
restart: unless-stopped
ports:
- "9090:9090"
volumes:
- ./monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- prometheus-data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--web.console.libraries=/etc/prometheus/console_libraries'
- '--web.console.templates=/etc/prometheus/consoles'
- '--storage.tsdb.retention.time=200h'
- '--web.enable-lifecycle'
networks:
- app-network
# Grafana仪表板
grafana:
image: grafana/grafana:latest
container_name: customer-service-grafana
restart: unless-stopped
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
volumes:
- grafana-data:/var/lib/grafana
- ./monitoring/grafana/dashboards:/etc/grafana/provisioning/dashboards:ro
- ./monitoring/grafana/datasources:/etc/grafana/provisioning/datasources:ro
networks:
- app-network
volumes:
redis-data:
postgres-data:
prometheus-data:
grafana-data:
networks:
app-network:
driver: bridge
4.2 Kubernetes云原生部署¶
4.2.1 Kubernetes部署清单¶
# k8s/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: customer-service
labels:
name: customer-service
---
# k8s/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: customer-service-config
namespace: customer-service
data:
ENVIRONMENT: "production"
REDIS_URL: "redis://redis-service:6379"
DATABASE_URL: "postgresql://user:password@postgres-service:5432/customer_service"
---
# k8s/secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: customer-service-secrets
namespace: customer-service
type: Opaque
data:
LANGSMITH_API_KEY: <base64-encoded-key>
OPENAI_API_KEY: <base64-encoded-key>
REDIS_PASSWORD: <base64-encoded-password>
POSTGRES_PASSWORD: <base64-encoded-password>
---
# k8s/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: customer-service-app
namespace: customer-service
labels:
app: customer-service
spec:
replicas: 3
selector:
matchLabels:
app: customer-service
template:
metadata:
labels:
app: customer-service
spec:
containers:
- name: customer-service
image: customer-service:latest
ports:
- containerPort: 8000
envFrom:
- configMapRef:
name: customer-service-config
- secretRef:
name: customer-service-secrets
resources:
requests:
memory: "1Gi"
cpu: "500m"
limits:
memory: "2Gi"
cpu: "1000m"
livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /ready
port: 8000
initialDelaySeconds: 5
periodSeconds: 5
volumeMounts:
- name: logs
mountPath: /app/logs
volumes:
- name: logs
persistentVolumeClaim:
claimName: customer-service-logs-pvc
---
# k8s/service.yaml
apiVersion: v1
kind: Service
metadata:
name: customer-service-svc
namespace: customer-service
spec:
selector:
app: customer-service
ports:
- protocol: TCP
port: 80
targetPort: 8000
type: ClusterIP
---
# k8s/hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: customer-service-hpa
namespace: customer-service
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: customer-service-app
minReplicas: 3
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80
4.2.2 Ingress和SSL配置¶
# k8s/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: customer-service-ingress
namespace: customer-service
annotations:
kubernetes.io/ingress.class: nginx
cert-manager.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/rate-limit: "100"
nginx.ingress.kubernetes.io/rate-limit-window: "1m"
spec:
tls:
- hosts:
- api.customer-service.com
secretName: customer-service-tls
rules:
- host: api.customer-service.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: customer-service-svc
port:
number: 80
4.3 云原生运维自动化¶
4.3.1 GitOps部署流水线¶
# .github/workflows/deploy.yml
name: Deploy to Production
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.11'
- name: Install dependencies
run: |
pip install -r requirements.txt
pip install pytest pytest-cov
- name: Run tests
run: |
pytest tests/ --cov=src/ --cov-report=xml
- name: Upload coverage
uses: codecov/codecov-action@v3
build:
needs: test
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/main'
steps:
- uses: actions/checkout@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Login to Container Registry
uses: docker/login-action@v2
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and push
uses: docker/build-push-action@v4
with:
context: .
push: true
tags: |
ghcr.io/${{ github.repository }}:latest
ghcr.io/${{ github.repository }}:${{ github.sha }}
cache-from: type=gha
cache-to: type=gha,mode=max
deploy:
needs: build
runs-on: ubuntu-latest
if: github.ref == 'refs/heads/main'
steps:
- uses: actions/checkout@v3
- name: Setup kubectl
uses: azure/setup-kubectl@v3
with:
version: 'v1.28.0'
- name: Deploy to Kubernetes
run: |
echo "${{ secrets.KUBECONFIG }}" | base64 -d > kubeconfig
export KUBECONFIG=kubeconfig
# 更新镜像标签
kubectl set image deployment/customer-service-app \
customer-service=ghcr.io/${{ github.repository }}:${{ github.sha }} \
-n customer-service
# 等待部署完成
kubectl rollout status deployment/customer-service-app -n customer-service
4.3.2 自动化运维脚本¶
# scripts/auto_ops.py - 自动化运维脚本
import asyncio
import logging
from typing import Dict, List
from kubernetes import client, config
from prometheus_client.parser import text_string_to_metric_families
import requests
class CloudNativeOpsManager:
"""云原生运维管理器"""
def __init__(self):
# 加载Kubernetes配置
try:
config.load_incluster_config() # 集群内配置
except:
config.load_kube_config() # 本地配置
self.k8s_apps_v1 = client.AppsV1Api()
self.k8s_core_v1 = client.CoreV1Api()
self.k8s_autoscaling_v2 = client.AutoscalingV2Api()
self.logger = logging.getLogger(__name__)
# 运维配置
self.namespace = "customer-service"
self.deployment_name = "customer-service-app"
self.prometheus_url = "http://prometheus:9090"
async def health_check(self) -> Dict[str, bool]:
"""系统健康检查"""
health_status = {}
try:
# 检查Pod状态
pods = self.k8s_core_v1.list_namespaced_pod(
namespace=self.namespace,
label_selector=f"app=customer-service"
)
healthy_pods = sum(1 for pod in pods.items
if pod.status.phase == "Running")
total_pods = len(pods.items)
health_status["pods"] = healthy_pods == total_pods and total_pods > 0
# 检查服务状态
services = self.k8s_core_v1.list_namespaced_service(
namespace=self.namespace
)
health_status["services"] = len(services.items) > 0
# 检查应用响应
try:
response = requests.get(
f"http://customer-service-svc.{self.namespace}/health",
timeout=10
)
health_status["application"] = response.status_code == 200
except:
health_status["application"] = False
self.logger.info(f"健康检查完成: {health_status}")
return health_status
except Exception as e:
self.logger.error(f"健康检查失败: {e}")
return {"error": True}
async def auto_scale_decision(self) -> Dict[str, any]:
"""自动扩缩容决策"""
try:
# 获取当前HPA状态
hpa = self.k8s_autoscaling_v2.read_namespaced_horizontal_pod_autoscaler(
name="customer-service-hpa",
namespace=self.namespace
)
current_replicas = hpa.status.current_replicas
desired_replicas = hpa.status.desired_replicas
# 获取性能指标
metrics = await self.get_performance_metrics()
# 扩缩容决策逻辑
decision = {
"current_replicas": current_replicas,
"desired_replicas": desired_replicas,
"cpu_usage": metrics.get("cpu_usage", 0),
"memory_usage": metrics.get("memory_usage", 0),
"request_rate": metrics.get("request_rate", 0),
"action": "none"
}
# 基于业务指标的智能扩缩容
if metrics.get("request_rate", 0) > 1000 and current_replicas < 8:
decision["action"] = "scale_up"
decision["reason"] = "High request rate detected"
elif metrics.get("cpu_usage", 0) > 80 and current_replicas < 10:
decision["action"] = "scale_up"
decision["reason"] = "High CPU usage"
elif (metrics.get("request_rate", 0) < 100 and
metrics.get("cpu_usage", 0) < 30 and
current_replicas > 3):
decision["action"] = "scale_down"
decision["reason"] = "Low resource usage"
return decision
except Exception as e:
self.logger.error(f"扩缩容决策失败: {e}")
return {"error": str(e)}
async def get_performance_metrics(self) -> Dict[str, float]:
"""获取性能指标"""
try:
metrics = {}
# CPU使用率
cpu_query = f'avg(rate(container_cpu_usage_seconds_total{{namespace="{self.namespace}"}}[5m])) * 100'
cpu_result = await self.query_prometheus(cpu_query)
metrics["cpu_usage"] = float(cpu_result[0]["value"][1]) if cpu_result else 0
# 内存使用率
memory_query = f'avg(container_memory_usage_bytes{{namespace="{self.namespace}"}}) / avg(container_spec_memory_limit_bytes{{namespace="{self.namespace}"}}) * 100'
memory_result = await self.query_prometheus(memory_query)
metrics["memory_usage"] = float(memory_result[0]["value"][1]) if memory_result else 0
# 请求速率
request_query = f'sum(rate(http_requests_total{{namespace="{self.namespace}"}}[5m]))'
request_result = await self.query_prometheus(request_query)
metrics["request_rate"] = float(request_result[0]["value"][1]) if request_result else 0
return metrics
except Exception as e:
self.logger.error(f"获取性能指标失败: {e}")
return {}
async def query_prometheus(self, query: str) -> List[Dict]:
"""查询Prometheus指标"""
try:
response = requests.get(
f"{self.prometheus_url}/api/v1/query",
params={"query": query},
timeout=10
)
data = response.json()
return data.get("data", {}).get("result", [])
except Exception as e:
self.logger.error(f"Prometheus查询失败: {e}")
return []
async def rolling_update(self, new_image: str):
"""滚动更新部署"""
try:
# 获取当前部署
deployment = self.k8s_apps_v1.read_namespaced_deployment(
name=self.deployment_name,
namespace=self.namespace
)
# 更新镜像
deployment.spec.template.spec.containers[0].image = new_image
# 应用更新
self.k8s_apps_v1.patch_namespaced_deployment(
name=self.deployment_name,
namespace=self.namespace,
body=deployment
)
self.logger.info(f"开始滚动更新到镜像: {new_image}")
# 等待更新完成
await self.wait_for_rollout()
except Exception as e:
self.logger.error(f"滚动更新失败: {e}")
raise
async def wait_for_rollout(self, timeout: int = 600):
"""等待滚动更新完成"""
import time
start_time = time.time()
while time.time() - start_time < timeout:
try:
deployment = self.k8s_apps_v1.read_namespaced_deployment(
name=self.deployment_name,
namespace=self.namespace
)
if (deployment.status.ready_replicas == deployment.spec.replicas and
deployment.status.updated_replicas == deployment.spec.replicas):
self.logger.info("滚动更新完成")
return True
await asyncio.sleep(10)
except Exception as e:
self.logger.error(f"检查滚动更新状态失败: {e}")
await asyncio.sleep(10)
raise TimeoutError("滚动更新超时")
# 运维主循环
async def ops_main_loop():
"""运维主循环"""
ops_manager = CloudNativeOpsManager()
while True:
try:
# 健康检查
health = await ops_manager.health_check()
if not all(health.values()):
logging.warning(f"系统健康检查异常: {health}")
# 自动扩缩容决策
scale_decision = await ops_manager.auto_scale_decision()
if scale_decision.get("action") != "none":
logging.info(f"扩缩容决策: {scale_decision}")
# 等待下一次检查
await asyncio.sleep(60) # 每分钟检查一次
except Exception as e:
logging.error(f"运维循环错误: {e}")
await asyncio.sleep(60)
if __name__ == "__main__":
logging.basicConfig(level=logging.INFO)
asyncio.run(ops_main_loop())
5. 实践练习¶
练习1:完整系统部署¶
- 使用Docker Compose部署完整的客服系统
- 配置Nginx负载均衡
- 设置SSL证书
- 验证系统功能
练习2:Kubernetes云原生部署¶
- 部署到Kubernetes集群
- 配置自动扩缩容
- 设置滚动更新
- 验证高可用性
练习3:监控告警配置¶
- 配置Prometheus监控指标
- 设置Grafana仪表板
- 配置告警规则
- 测试告警机制
练习4:性能优化实战¶
- 进行压力测试
- 识别性能瓶颈
- 实施优化方案
- 验证优化效果
课程总结¶
关键收获¶
- 完整开发流程:从需求分析到生产部署的全流程实践
- 企业级架构:可扩展、高可用的系统架构设计
- 运维最佳实践:监控、日志、性能优化的实战经验
- 问题解决能力:具备独立解决复杂技术问题的能力
后续学习建议¶
- 深入学习云原生技术(Kubernetes、Service Mesh)
- 探索更多AI技术(多模态、强化学习)
- 关注行业最新发展(AGI、具身智能)
- 参与开源项目,积累实战经验
认证考核¶
- 完成综合项目实战
- 通过技术答辩
- 获得企业级多智能体系统开发认证
恭喜您完成了多智能体AI系统培训的全部课程!