第三天:LangSmith监控平台集成¶
学习目标¶
- 掌握LangSmith平台的核心功能和架构
- 学会集成LangSmith到多智能体系统
- 掌握系统监控、日志分析和性能优化技术
- 学会使用LangSmith进行调试和故障排除
- 了解企业级监控系统的设计和实现
- 掌握智能告警和自动化运维技术
参考项目¶
本课程深入讲解LangSmith监控平台的集成与应用,帮助学员掌握多智能体系统的监控、调试和性能优化技术。
💡 实际代码参考:完整的LangSmith监控集成实现可参考项目中的以下文件:
langsmith_integration.py- 企业级全链路追踪系统src/monitoring/langsmith_integration.py- 系统性能监控src/monitoring/langsmith_integration.py- 智能告警管理multi_agent_system/src/monitoring/- 监控组件集合src/examples/customer_service_system.py- 客服系统监控案例
模拟环境说明:本培训使用模拟的LangSmith环境进行教学,所有功能和API接口与真实LangSmith保持一致。在实际项目中,只需要替换为真实的LangSmith API密钥即可无缝切换到生产环境。
代码引用: 完整的企业级监控系统实现请参考 multi_agent_system/src/monitoring/langsmith_integration.py
企业级监控系统的核心特性:
核心组件:
AlertLevel: 告警级别管理(信息、警告、错误、严重)MetricType: 指标类型分类(性能、业务、系统、安全)MonitoringConfig: 监控配置管理(采样率、批处理、保留策略)EnterpriseMonitoringSystem: 主监控系统,支持全链路追踪
企业级特性:
- 智能体执行全链路追踪
- 实时告警规则检查
- 性能指标自动收集
- 可配置的数据保留策略
1. LangSmith平台概述¶
1.1 什么是LangSmith¶
LangSmith是LangChain生态系统中的企业级监控和调试平台,专为LLM应用和多智能体系统设计。它提供了完整的可观测性解决方案,帮助开发者更好地理解和优化他们的AI应用。
1.2 核心价值¶
- 全链路追踪:完整记录智能体交互过程,包括输入输出和中间步骤
- 性能监控:实时监控系统性能指标,如响应时间、成功率等
- 调试支持:提供强大的调试和故障排除工具,快速定位问题
- 数据分析:深入分析智能体行为和系统效率,支持数据驱动的优化
- 成本控制:监控Token使用量和API调用成本
- 企业级特性:支持多租户、权限管理、数据安全等企业需求
1.3 企业级架构组件¶
# 基于项目 multi_agent_system/src/monitoring/langsmith_integration.py 的架构设计
from typing import Dict, Any, List, Optional
from dataclasses import dataclass
from datetime import datetime
import uuid
@dataclass
class TracingContext:
"""追踪上下文"""
trace_id: str
span_id: str
parent_span_id: Optional[str]
operation_name: str
start_time: datetime
tags: Dict[str, Any]
metadata: Dict[str, Any]
@dataclass
class RunMetrics:
"""运行指标"""
run_id: str
duration_ms: float
token_usage: int
cost_usd: float
memory_usage_mb: float
cpu_usage_percent: float
success_rate: float
error_count: int
class EnterpriseTracing:
"""企业级全链路追踪系统"""
def __init__(self, project_name: str, environment: str = "production"):
self.project_name = project_name
self.environment = environment
self.active_traces: Dict[str, TracingContext] = {}
self.metrics_buffer: List[RunMetrics] = []
self.alert_thresholds = {
"error_rate": 0.05, # 5% 错误率阈值
"response_time": 5000, # 5秒响应时间阈值
"memory_usage": 80, # 80% 内存使用率阈值
}
def start_trace(self, operation_name: str,
parent_span_id: Optional[str] = None,
tags: Dict[str, Any] = None) -> str:
"""开始追踪"""
trace_id = str(uuid.uuid4())
span_id = str(uuid.uuid4())
context = TracingContext(
trace_id=trace_id,
span_id=span_id,
parent_span_id=parent_span_id,
operation_name=operation_name,
start_time=datetime.now(),
tags=tags or {},
metadata={}
)
self.active_traces[trace_id] = context
return trace_id
def end_trace(self, trace_id: str,
outputs: Dict[str, Any] = None,
error: Optional[str] = None) -> RunMetrics:
"""结束追踪并生成指标"""
if trace_id not in self.active_traces:
raise ValueError(f"Trace {trace_id} not found")
context = self.active_traces[trace_id]
end_time = datetime.now()
duration_ms = (end_time - context.start_time).total_seconds() * 1000
# 生成运行指标
metrics = RunMetrics(
run_id=trace_id,
duration_ms=duration_ms,
token_usage=self._calculate_token_usage(outputs),
cost_usd=self._calculate_cost(outputs),
memory_usage_mb=self._get_memory_usage(),
cpu_usage_percent=self._get_cpu_usage(),
success_rate=1.0 if error is None else 0.0,
error_count=1 if error else 0
)
self.metrics_buffer.append(metrics)
# 检查告警阈值
self._check_alerts(metrics)
# 清理追踪上下文
del self.active_traces[trace_id]
return metrics
1.4 模拟环境与生产环境对比¶
| 特性 | 模拟环境 | 生产环境 | 说明 |
|---|---|---|---|
| API接口 | 完全兼容 | 真实API | 接口设计完全一致 |
| 数据存储 | 内存存储 | 云端存储 | 模拟环境数据重启后清空 |
| 性能指标 | 模拟数据 | 真实数据 | 算法和逻辑完全相同 |
| 告警系统 | 本地日志 | 多渠道告警 | 支持邮件、短信、钉钉等 |
| 数据可视化 | 简化版本 | 完整仪表板 | 核心图表和指标一致 |
| 权限管理 | 简化版本 | 企业级RBAC | 支持多租户和细粒度权限 |
🚀 快速切换到生产环境:
# 开发/测试环境 tracer = EnterpriseTracing( project_name="my-project", environment="development", api_key="mock-key" # 使用模拟密钥 ) # 生产环境 tracer = EnterpriseTracing( project_name="my-project", environment="production", api_key=os.getenv("LANGSMITH_API_KEY") # 使用真实密钥 )
2. 核心功能详解¶
2.1 链路追踪(Tracing)¶
2.1.1 基本概念¶
from langsmith import Client
from langchain.callbacks import LangChainTracer
# 初始化LangSmith客户端
client = Client(
api_url="https://api.smith.langchain.com",
api_key="your-api-key"
)
# 配置追踪器
tracer = LangChainTracer(
project_name="multi-agent-system",
client=client
)
2.1.2 智能体追踪实现¶
from langchain.schema import BaseMessage
from typing import List, Dict, Any
class TracedAgent:
def __init__(self, name: str, tracer: LangChainTracer):
self.name = name
self.tracer = tracer
def process_message(self, message: str, context: Dict[str, Any] = None):
"""带追踪的消息处理"""
with self.tracer.trace(
name=f"{self.name}_process",
inputs={"message": message, "context": context}
) as trace:
try:
# 模拟智能体处理逻辑
result = self._internal_process(message, context)
trace.outputs = {"result": result}
return result
except Exception as e:
trace.error = str(e)
raise
def _internal_process(self, message: str, context: Dict[str, Any]):
# 实际处理逻辑
return f"Processed: {message}"
2.2 日志管理¶
2.2.1 结构化日志¶
import logging
from langsmith import Client
from datetime import datetime
class LangSmithLogger:
def __init__(self, project_name: str):
self.client = Client()
self.project_name = project_name
def log_agent_action(self, agent_name: str, action: str,
inputs: Dict, outputs: Dict, metadata: Dict = None):
"""记录智能体行为"""
log_entry = {
"timestamp": datetime.utcnow().isoformat(),
"agent_name": agent_name,
"action": action,
"inputs": inputs,
"outputs": outputs,
"metadata": metadata or {}
}
self.client.create_run(
name=f"{agent_name}_{action}",
project_name=self.project_name,
inputs=inputs,
outputs=outputs,
extra=metadata
)
2.3 性能指标收集¶
2.3.1 关键指标定义¶
from dataclasses import dataclass
from typing import Optional
import time
@dataclass
class PerformanceMetrics:
agent_name: str
operation: str
start_time: float
end_time: Optional[float] = None
duration: Optional[float] = None
memory_usage: Optional[float] = None
token_count: Optional[int] = None
success: bool = True
error_message: Optional[str] = None
class MetricsCollector:
def __init__(self, langsmith_client: Client):
self.client = langsmith_client
self.metrics = []
def start_operation(self, agent_name: str, operation: str) -> str:
"""开始操作计时"""
run_id = f"{agent_name}_{operation}_{int(time.time())}"
metric = PerformanceMetrics(
agent_name=agent_name,
operation=operation,
start_time=time.time()
)
self.metrics.append(metric)
return run_id
def end_operation(self, run_id: str, success: bool = True,
error_message: str = None):
"""结束操作计时"""
# 查找对应的指标记录并更新
for metric in self.metrics:
if f"{metric.agent_name}_{metric.operation}" in run_id:
metric.end_time = time.time()
metric.duration = metric.end_time - metric.start_time
metric.success = success
metric.error_message = error_message
# 发送到LangSmith
self._send_metrics(metric)
break
def _send_metrics(self, metric: PerformanceMetrics):
"""发送指标到LangSmith"""
self.client.create_run(
name=f"metrics_{metric.agent_name}_{metric.operation}",
inputs={"operation": metric.operation},
outputs={
"duration": metric.duration,
"success": metric.success,
"memory_usage": metric.memory_usage,
"token_count": metric.token_count
},
extra={
"agent_name": metric.agent_name,
"error_message": metric.error_message
}
)
3. 集成实现¶
3.1 LangGraph + LangSmith集成¶
3.1.1 基础集成配置¶
from langgraph import StateGraph
from langsmith import Client
from langchain.callbacks import LangChainTracer
import os
# 环境配置
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_ENDPOINT"] = "https://api.smith.langchain.com"
os.environ["LANGCHAIN_API_KEY"] = "your-api-key"
os.environ["LANGCHAIN_PROJECT"] = "multi-agent-system"
class MonitoredMultiAgentSystem:
def __init__(self):
self.client = Client()
self.tracer = LangChainTracer(project_name="multi-agent-system")
self.graph = self._build_graph()
def _build_graph(self):
"""构建带监控的智能体图"""
graph = StateGraph(dict)
# 添加节点(智能体)
graph.add_node("coordinator", self._coordinator_with_monitoring)
graph.add_node("analyzer", self._analyzer_with_monitoring)
graph.add_node("executor", self._executor_with_monitoring)
# 定义边
graph.add_edge("coordinator", "analyzer")
graph.add_edge("analyzer", "executor")
graph.add_edge("executor", "coordinator")
# 设置入口点
graph.set_entry_point("coordinator")
return graph.compile()
def _coordinator_with_monitoring(self, state: dict):
"""带监控的协调器智能体"""
with self.tracer.trace(
name="coordinator_process",
inputs=state
) as trace:
try:
# 协调器逻辑
result = self._coordinate_tasks(state)
trace.outputs = result
return result
except Exception as e:
trace.error = str(e)
raise
def _analyzer_with_monitoring(self, state: dict):
"""带监控的分析器智能体"""
with self.tracer.trace(
name="analyzer_process",
inputs=state
) as trace:
try:
# 分析器逻辑
result = self._analyze_data(state)
trace.outputs = result
return result
except Exception as e:
trace.error = str(e)
raise
def _executor_with_monitoring(self, state: dict):
"""带监控的执行器智能体"""
with self.tracer.trace(
name="executor_process",
inputs=state
) as trace:
try:
# 执行器逻辑
result = self._execute_tasks(state)
trace.outputs = result
return result
except Exception as e:
trace.error = str(e)
raise
3.2 自定义监控装饰器¶
3.2.1 智能体监控装饰器¶
from functools import wraps
from typing import Callable, Any
import inspect
def monitor_agent(agent_name: str, operation: str = None):
"""智能体监控装饰器"""
def decorator(func: Callable) -> Callable:
@wraps(func)
def wrapper(*args, **kwargs) -> Any:
# 获取操作名称
op_name = operation or func.__name__
# 开始追踪
with LangChainTracer().trace(
name=f"{agent_name}_{op_name}",
inputs={"args": args, "kwargs": kwargs}
) as trace:
try:
# 执行原函数
result = func(*args, **kwargs)
trace.outputs = {"result": result}
return result
except Exception as e:
trace.error = str(e)
raise
return wrapper
return decorator
# 使用示例
class SmartAgent:
@monitor_agent("smart_agent", "process_request")
def process_request(self, request: str) -> str:
"""处理请求"""
# 实际处理逻辑
return f"Processed: {request}"
@monitor_agent("smart_agent", "analyze_data")
def analyze_data(self, data: dict) -> dict:
"""分析数据"""
# 分析逻辑
return {"analysis": "completed", "insights": ["insight1", "insight2"]}
4. 监控与分析¶
4.1 实时监控面板¶
4.1.1 关键指标监控¶
from langsmith import Client
from typing import List, Dict
import asyncio
class RealTimeMonitor:
def __init__(self, project_name: str):
self.client = Client()
self.project_name = project_name
self.is_monitoring = False
async def start_monitoring(self, interval: int = 30):
"""开始实时监控"""
self.is_monitoring = True
while self.is_monitoring:
try:
metrics = await self._collect_metrics()
await self._analyze_metrics(metrics)
await asyncio.sleep(interval)
except Exception as e:
print(f"监控错误: {e}")
async def _collect_metrics(self) -> Dict:
"""收集系统指标"""
runs = self.client.list_runs(
project_name=self.project_name,
limit=100
)
metrics = {
"total_runs": len(runs),
"success_rate": 0,
"avg_duration": 0,
"error_count": 0,
"agent_performance": {}
}
if runs:
successful_runs = [r for r in runs if not r.error]
metrics["success_rate"] = len(successful_runs) / len(runs)
durations = [r.total_time for r in runs if r.total_time]
if durations:
metrics["avg_duration"] = sum(durations) / len(durations)
metrics["error_count"] = len(runs) - len(successful_runs)
return metrics
async def _analyze_metrics(self, metrics: Dict):
"""分析指标并生成告警"""
# 性能告警
if metrics["avg_duration"] > 5.0: # 5秒阈值
await self._send_alert("性能告警", f"平均响应时间过长: {metrics['avg_duration']:.2f}s")
# 错误率告警
if metrics["success_rate"] < 0.95: # 95%成功率阈值
await self._send_alert("错误率告警", f"成功率过低: {metrics['success_rate']:.2%}")
async def _send_alert(self, alert_type: str, message: str):
"""发送告警"""
print(f"[{alert_type}] {message}")
# 这里可以集成邮件、Slack等告警渠道
4.2 数据分析与洞察¶
4.2.1 智能体行为分析¶
import pandas as pd
import matplotlib.pyplot as plt
from typing import List, Dict
class AgentAnalytics:
def __init__(self, langsmith_client: Client):
self.client = langsmith_client
def analyze_agent_performance(self, agent_name: str,
time_range: int = 24) -> Dict:
"""分析智能体性能"""
# 获取指定时间范围内的运行数据
runs = self.client.list_runs(
project_name="multi-agent-system",
filter=f"name LIKE '%{agent_name}%'",
limit=1000
)
if not runs:
return {"error": "没有找到相关数据"}
# 转换为DataFrame进行分析
data = []
for run in runs:
data.append({
"timestamp": run.start_time,
"duration": run.total_time or 0,
"success": not bool(run.error),
"tokens": run.prompt_tokens + run.completion_tokens if hasattr(run, 'prompt_tokens') else 0,
"operation": run.name
})
df = pd.DataFrame(data)
# 计算关键指标
analysis = {
"total_operations": len(df),
"success_rate": df["success"].mean(),
"avg_duration": df["duration"].mean(),
"max_duration": df["duration"].max(),
"min_duration": df["duration"].min(),
"total_tokens": df["tokens"].sum(),
"operations_per_hour": self._calculate_ops_per_hour(df),
"performance_trend": self._calculate_trend(df)
}
return analysis
def _calculate_ops_per_hour(self, df: pd.DataFrame) -> float:
"""计算每小时操作数"""
if df.empty:
return 0
time_span = (df["timestamp"].max() - df["timestamp"].min()).total_seconds() / 3600
return len(df) / max(time_span, 1)
def _calculate_trend(self, df: pd.DataFrame) -> str:
"""计算性能趋势"""
if len(df) < 10:
return "数据不足"
# 按时间排序并计算移动平均
df_sorted = df.sort_values("timestamp")
df_sorted["duration_ma"] = df_sorted["duration"].rolling(window=10).mean()
# 比较最近和早期的平均值
recent_avg = df_sorted["duration_ma"].tail(10).mean()
early_avg = df_sorted["duration_ma"].head(10).mean()
if recent_avg > early_avg * 1.1:
return "性能下降"
elif recent_avg < early_avg * 0.9:
return "性能提升"
else:
return "性能稳定"
5. 性能优化¶
5.1 基于监控数据的优化¶
5.1.1 自动优化建议¶
class PerformanceOptimizer:
def __init__(self, analytics: AgentAnalytics):
self.analytics = analytics
def generate_optimization_suggestions(self, agent_name: str) -> List[Dict]:
"""生成优化建议"""
analysis = self.analytics.analyze_agent_performance(agent_name)
suggestions = []
# 响应时间优化
if analysis["avg_duration"] > 3.0:
suggestions.append({
"type": "performance",
"priority": "high",
"issue": "响应时间过长",
"suggestion": "考虑优化模型调用、增加缓存或并行处理",
"expected_improvement": "30-50%响应时间减少"
})
# 成功率优化
if analysis["success_rate"] < 0.95:
suggestions.append({
"type": "reliability",
"priority": "critical",
"issue": "成功率过低",
"suggestion": "增加错误处理、重试机制和输入验证",
"expected_improvement": "成功率提升至98%+"
})
# 资源使用优化
if analysis["total_tokens"] > 100000:
suggestions.append({
"type": "cost",
"priority": "medium",
"issue": "Token使用量过高",
"suggestion": "优化提示词、使用更小的模型或实现智能缓存",
"expected_improvement": "20-40%成本降低"
})
return suggestions
def implement_caching_optimization(self, agent_class):
"""实现缓存优化"""
from functools import lru_cache
import hashlib
class CachedAgent(agent_class):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self._cache = {}
def _get_cache_key(self, inputs: Dict) -> str:
"""生成缓存键"""
content = str(sorted(inputs.items()))
return hashlib.md5(content.encode()).hexdigest()
def process_with_cache(self, inputs: Dict, ttl: int = 3600):
"""带缓存的处理"""
cache_key = self._get_cache_key(inputs)
# 检查缓存
if cache_key in self._cache:
cached_result, timestamp = self._cache[cache_key]
if time.time() - timestamp < ttl:
return cached_result
# 执行处理
result = self.process(inputs)
# 存储到缓存
self._cache[cache_key] = (result, time.time())
return result
return CachedAgent
5.2 智能负载均衡¶
5.2.1 基于性能的负载分配¶
import heapq
from typing import List, Dict, Any
class IntelligentLoadBalancer:
def __init__(self, agents: List[Any]):
self.agents = agents
self.agent_metrics = {id(agent): {"load": 0, "avg_response_time": 0}
for agent in agents}
def select_optimal_agent(self, task_complexity: float = 1.0) -> Any:
"""选择最优智能体"""
# 计算每个智能体的负载分数
scores = []
for agent in self.agents:
agent_id = id(agent)
metrics = self.agent_metrics[agent_id]
# 综合考虑负载和响应时间
score = (metrics["load"] * 0.6 +
metrics["avg_response_time"] * 0.4) * task_complexity
heapq.heappush(scores, (score, agent))
# 返回负载最低的智能体
_, optimal_agent = heapq.heappop(scores)
return optimal_agent
def update_agent_metrics(self, agent: Any, response_time: float):
"""更新智能体指标"""
agent_id = id(agent)
metrics = self.agent_metrics[agent_id]
# 更新平均响应时间(指数移动平均)
alpha = 0.3
metrics["avg_response_time"] = (
alpha * response_time +
(1 - alpha) * metrics["avg_response_time"]
)
# 更新负载(简单递减)
metrics["load"] = max(0, metrics["load"] - 0.1)
def assign_task(self, task: Dict[str, Any]) -> Any:
"""分配任务"""
# 评估任务复杂度
complexity = self._evaluate_task_complexity(task)
# 选择最优智能体
agent = self.select_optimal_agent(complexity)
# 增加负载
agent_id = id(agent)
self.agent_metrics[agent_id]["load"] += complexity
return agent
def _evaluate_task_complexity(self, task: Dict[str, Any]) -> float:
"""评估任务复杂度"""
# 简单的复杂度评估逻辑
base_complexity = 1.0
# 根据任务类型调整
if task.get("type") == "analysis":
base_complexity *= 1.5
elif task.get("type") == "generation":
base_complexity *= 2.0
# 根据数据量调整
data_size = len(str(task.get("data", "")))
if data_size > 1000:
base_complexity *= 1.2
return base_complexity
6. 实践练习¶
练习1:基础监控集成¶
目标:为现有的多智能体系统集成LangSmith监控
任务:
- 配置LangSmith环境
- 为智能体添加基础追踪
- 实现日志收集
- 创建简单的监控面板
代码框架:
# 学员需要完成的代码
class MonitoredAgent:
def __init__(self, name: str):
self.name = name
# TODO: 初始化LangSmith客户端和追踪器
def process_task(self, task: str):
# TODO: 添加监控和追踪逻辑
pass
def get_performance_metrics(self):
# TODO: 获取性能指标
pass
练习2:性能分析与优化¶
目标:分析智能体性能并实现优化
任务:
- 收集性能数据
- 分析瓶颈
- 实现缓存机制
- 验证优化效果
练习3:告警系统实现¶
目标:构建智能告警系统
任务:
- 定义告警规则
- 实现告警触发逻辑
- 集成通知渠道
- 测试告警功能
总结¶
本课程涵盖了LangSmith监控平台的核心功能和集成方法,通过实践练习帮助学员掌握:
- 监控集成:完整的LangSmith集成流程
- 性能分析:深入的性能分析和优化技术
- 实时监控:构建实时监控和告警系统
- 最佳实践:企业级监控的最佳实践
下一课程将学习企业级系统架构设计与实现。