Browse Source

feat(eval): 第一方 agent 行为回归框架(评估建议②) — 让智能体质量可度量

有 pipeline/gate 单测、缺 agent 行为回归集。新增 eval 闭环:
- agentpaas/engine/agent_eval.py:EvalTask/EvalResult/run_eval_task/summarize。
  "跑 agent / 收产物 / judge" 全可注入 → 框架可 mock 单测(进 CI);复用 pipeline
  的 default_judge。量化:落盘达标(must_produce)、LLM-judge 产物分、步数/成本、通过率。
- evals/golden/*.yaml:literature-mapper / assignment-grader 的 golden 任务。
- evals/run_evals.py:live runner(POST /run → 收 workspace 产物 → default_judge →
  汇总 + last_report.json),手动/nightly 触发(要 AGENTPAAS_API_KEY、花钱)。
- 回归 tests/test_agent_eval.py(7)+ agentpaas 351 全绿。
- **live 验证**:default_judge 对真实 literature_map.md 打 0.90 分、passed(ollama)。

下一步可用它量化 FC vs 文本路径,数据驱动决定 nativeToolCalls 是否设默认(阶段3)。

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
kenny67nju 5 months ago
parent
commit
84e456176c

+ 102 - 0
agentpaas/src/agentpaas/engine/agent_eval.py

@@ -0,0 +1,102 @@
+"""
+agentpaas.engine.agent_eval — 第一方 agent 的"行为回归"评估框架。
+
+评估建议②(docs/AGENT_QUALITY_ASSESSMENT.md):有 pipeline/gate 单测,缺第一方 agent
+的行为回归集——"真实智能体质量"没被覆盖。本框架给每个 agent 配 golden 任务,跑完后量化:
+落盘达标(must_produce)、LLM-judge 产物分、步数/成本。复用 pipeline 的 default_judge。
+
+设计要点:run_eval_task 的"跑 agent / 收产物 / judge"全部**可注入** → 框架本身可用 mock
+单测(进 CI);真实 LLM 跑由 evals/run_evals.py 手动/nightly 触发(要 API key、花钱)。
+"""
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Callable, Dict, List, Optional
+
+
+@dataclass
+class EvalTask:
+    """一个 golden 评估任务。"""
+    id: str
+    agent_id: str = ""                       # 跑哪个 agent 实例
+    agent_template: str = ""                 # 或按模板(runner 解析成实例)
+    input: str = ""
+    must_produce: List[str] = field(default_factory=list)  # 期望产物文件名(子串匹配)
+    rubric: str = ""                         # LLM-judge 评分标准(空则跳过 judge)
+    threshold: float = 0.6
+    native_tool_calls: Optional[bool] = None  # 临时覆盖 react.nativeToolCalls(对比 FC vs 文本)
+
+
+@dataclass
+class EvalResult:
+    task_id: str
+    produced_files: List[str] = field(default_factory=list)
+    files_ok: bool = True                    # must_produce 全部命中
+    missing_files: List[str] = field(default_factory=list)
+    judge_score: float = 0.0
+    judge_passed: bool = True
+    judge_reasons: List[str] = field(default_factory=list)
+    steps: int = 0
+    cost_usd: float = 0.0
+    error: str = ""
+    passed: bool = False                     # 总判定 = files_ok 且 judge_passed 且无 error
+
+
+def run_eval_task(task: EvalTask, *,
+                  run_fn: Callable[[EvalTask], dict],
+                  collect_artifacts_fn: Callable[[str], Dict[str, str]],
+                  judge_fn: Callable) -> EvalResult:
+    """跑一个 golden 任务并打分。三个回调全可注入:
+      run_fn(task) -> {"output", "steps", "cost_usd", "workspace_path"}
+      collect_artifacts_fn(workspace_path) -> {filename: text}
+      judge_fn(JudgeRequest) -> JudgeResult
+    """
+    try:
+        run = run_fn(task)
+    except Exception as e:  # 跑挂了 = 这条任务失败,不是框架崩
+        return EvalResult(task_id=task.id, error=str(e)[:400], passed=False)
+
+    files = collect_artifacts_fn(run.get("workspace_path", "")) or {}
+    names = list(files.keys())
+    missing = [m for m in task.must_produce
+               if not any(m in n for n in names)]
+    files_ok = not missing
+
+    judge_score, judge_passed, reasons = 0.0, True, []
+    if task.rubric:
+        from agentpaas.engine.pipeline import JudgeRequest
+        artifacts_text = "\n\n".join(f"## {n}\n{t[:4000]}" for n, t in files.items())
+        if not artifacts_text:
+            artifacts_text = str(run.get("output", ""))[:8000]
+        jr = judge_fn(JudgeRequest(rubric=task.rubric, threshold=task.threshold,
+                                   model="", artifacts_text=artifacts_text, stage_id=task.id))
+        judge_score, judge_passed, reasons = jr.score, jr.passed, list(jr.reasons or [])
+
+    err = str(run.get("output", "")) if _looks_like_error(run.get("output", "")) else ""
+    passed = files_ok and judge_passed and not err
+    return EvalResult(
+        task_id=task.id, produced_files=names, files_ok=files_ok, missing_files=missing,
+        judge_score=judge_score, judge_passed=judge_passed, judge_reasons=reasons,
+        steps=int(run.get("steps", 0) or 0), cost_usd=float(run.get("cost_usd", 0.0) or 0.0),
+        error=err, passed=passed,
+    )
+
+
+def _looks_like_error(output: str) -> bool:
+    import re
+    return bool(re.search(r"\[[A-Z0-9_-]*(ERROR|TIMEOUT)\]", str(output or "")))
+
+
+def summarize(results: List[EvalResult]) -> dict:
+    """汇总:通过率、落盘达标率、平均分/步数/成本。"""
+    n = len(results) or 1
+    return {
+        "total": len(results),
+        "passed": sum(1 for r in results if r.passed),
+        "pass_rate": round(sum(1 for r in results if r.passed) / n, 3),
+        "files_ok_rate": round(sum(1 for r in results if r.files_ok) / n, 3),
+        "avg_judge_score": round(sum(r.judge_score for r in results) / n, 3),
+        "avg_steps": round(sum(r.steps for r in results) / n, 1),
+        "total_cost_usd": round(sum(r.cost_usd for r in results), 4),
+        "failures": [r.task_id for r in results if not r.passed],
+    }

+ 1 - 1
docs/AGENT_QUALITY_ASSESSMENT.md

@@ -65,7 +65,7 @@ dir / session / MCP / 401 / context / network 反应式逐个修。根因:reac
 > sandbox + 参数 canonicalization 的**工具层硬约束**问题(`file_tools._resolve`/`_sandbox`,
 > commit 5f38a1c/3bae21d/df648aa 已部分解决)。FC 与工具层约束是两件事,别把后者算进 FC 收益。
 
-### ② 建第一方 agent 的行为回归集 —— 让质量可度量
+### ② 建第一方 agent 的行为回归集 —— 让质量可度量【已建框架, commit 见下】
 - **修正认知**:不是"没有 eval"。`tests/test_pipeline.py` 已有 LLM-judge gate、阈值、halt、
   成本进 term graph——**eval 基础设施长出来了一截**。真正的缺口是:**有 pipeline/gate 单测,
   没有第一方 agent 的"行为回归集"**(真实智能体质量没被覆盖)。

+ 10 - 0
evals/golden/assignment-grader.yaml

@@ -0,0 +1,10 @@
+# 作业批改助手 golden 任务(需先放好作业要求+提交物,这里是占位骨架)
+agent_template: teaching.assignment-grader
+tasks:
+  - id: grade-sample
+    input: "批改 data/ 下的学生提交,按作业要求生成评阅报告。"
+    must_produce: ["review"]
+    rubric: |
+      评估这份批改报告(0-1):是否按 rubric 两轴(工程产物/AI协作)逐项打分、
+      每分是否锚定提交物中的证据、是否先判红线、总评是否给出可执行建议。>=0.6 合格。
+    threshold: 0.6

+ 13 - 0
evals/golden/literature-mapper.yaml

@@ -0,0 +1,13 @@
+# 文献地图助手 golden 任务
+agent_template: research.literature-mapper
+tasks:
+  - id: sla-lit-map
+    input: "我的研究方向是二语习得,请帮我梳理文献地图。"
+    must_produce: ["literature_map", "references"]
+    rubric: |
+      评估这份"二语习得文献地图"的质量(0-1):
+      (a) 是否覆盖核心理论框架(输入假说/输出假说/互动假说/普遍语法/社会文化/注意假说等);
+      (b) 每个框架是否给出代表人物 + 核心观点 + 关键文献;
+      (c) 结构是否清晰、可直接用于开题。
+      给 0-1 分,>=0.6 视为合格。
+    threshold: 0.6

+ 126 - 0
evals/run_evals.py

@@ -0,0 +1,126 @@
+#!/usr/bin/env python3
+"""第一方 agent 行为回归 — live 评估 runner(手动/nightly,要真 LLM、花钱)。
+
+用法:
+  AGENTPAAS_API_KEY=<key> python3 evals/run_evals.py [golden/xxx.yaml ...]
+不传文件则跑 evals/golden/ 下全部。结果打印汇总 + 落 evals/last_report.json。
+
+框架(打分逻辑)在 agentpaas.engine.agent_eval(已单测,进 CI);本 runner 只负责
+把真实"跑 agent / 收产物 / judge"接进去。judge 复用 pipeline.default_judge。
+"""
+from __future__ import annotations
+
+import glob
+import json
+import os
+import sys
+import time
+import urllib.request
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "agentpaas", "src"))
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "lambdagent", "src"))
+
+from agentpaas.engine.agent_eval import EvalTask, run_eval_task, summarize  # noqa: E402
+
+BASE = os.environ.get("AGENTPAAS_URL", "http://127.0.0.1:8000")
+KEY = os.environ.get("AGENTPAAS_API_KEY", "")
+
+
+def _api(method, path, body=None):
+    url = f"{BASE}{path}"
+    data = json.dumps(body).encode() if body is not None else None
+    req = urllib.request.Request(url, data=data, method=method,
+                                 headers={"Content-Type": "application/json",
+                                          "Authorization": f"Bearer {KEY}"})
+    with urllib.request.urlopen(req, timeout=1800) as r:
+        return json.loads(r.read())
+
+
+def _resolve_agent_id(template: str) -> str:
+    """按模板找一个已安装实例(取第一个 active)。"""
+    agents = _api("GET", "/api/v1/agents").get("agents", [])
+    for a in agents:
+        if a.get("agent_template") == template and a.get("status", "active") == "active":
+            return a["id"]
+    raise RuntimeError(f"未找到模板 {template} 的已安装实例")
+
+
+def _run_fn(task: EvalTask) -> dict:
+    aid = task.agent_id or _resolve_agent_id(task.agent_template)
+    # 同步 /run:返回时已完成
+    res = _api("POST", f"/api/v1/agents/{aid}/run", {"input": task.input})
+    return {
+        "output": res.get("output", ""),
+        "steps": (res.get("usage") or {}).get("steps", res.get("steps", 0)),
+        "cost_usd": res.get("cost_usd", 0.0),
+        "workspace_path": res.get("workspace_path", ""),
+        "_run_id": res.get("id", ""), "_agent_id": aid,
+    }
+
+
+def _collect_fn(workspace_path: str) -> dict:
+    # 直接读磁盘(workspace_path 是本机绝对路径)
+    files = {}
+    if not workspace_path or not os.path.isdir(workspace_path):
+        return files
+    skip = {"config.yml", "cost.json", "input.json", "manifest.json", "output.json", "trace.json"}
+    for dp, _dn, fn in os.walk(workspace_path):
+        for f in fn:
+            if f in skip or f.startswith("."):
+                continue
+            full = os.path.join(dp, f)
+            try:
+                with open(full, encoding="utf-8", errors="ignore") as fh:
+                    files[os.path.relpath(full, workspace_path)] = fh.read()
+            except Exception:
+                pass
+    return files
+
+
+def _judge_fn(req):
+    from agentpaas.engine.pipeline import default_judge
+    return default_judge(req)
+
+
+def _load_tasks(paths):
+    import yaml
+    tasks = []
+    for p in paths:
+        spec = yaml.safe_load(open(p, encoding="utf-8"))
+        tmpl = spec.get("agent_template", "")
+        for t in spec.get("tasks", []):
+            tasks.append(EvalTask(
+                id=t["id"], agent_template=tmpl, agent_id=t.get("agent_id", ""),
+                input=t.get("input", ""), must_produce=t.get("must_produce", []),
+                rubric=t.get("rubric", ""), threshold=float(t.get("threshold", 0.6))))
+    return tasks
+
+
+def main():
+    if not KEY:
+        print("✗ 需要 AGENTPAAS_API_KEY 环境变量(同前端用的那把)。"); return 2
+    paths = sys.argv[1:] or sorted(glob.glob(os.path.join(os.path.dirname(__file__), "golden", "*.yaml")))
+    tasks = _load_tasks(paths)
+    print(f"跑 {len(tasks)} 个 golden 任务 @ {BASE}\n")
+    results = []
+    for t in tasks:
+        t0 = time.time()
+        r = run_eval_task(t, run_fn=_run_fn, collect_artifacts_fn=_collect_fn, judge_fn=_judge_fn)
+        results.append(r)
+        flag = "✅" if r.passed else "❌"
+        print(f"{flag} {t.id:24} files_ok={r.files_ok} judge={r.judge_score:.2f} "
+              f"steps={r.steps} ${r.cost_usd:.3f} {int(time.time()-t0)}s"
+              + (f"  缺:{r.missing_files}" if r.missing_files else "")
+              + (f"  err:{r.error[:60]}" if r.error else ""))
+    s = summarize(results)
+    print("\n=== 汇总 ===")
+    print(json.dumps(s, ensure_ascii=False, indent=2))
+    out = os.path.join(os.path.dirname(__file__), "last_report.json")
+    json.dump({"summary": s, "results": [r.__dict__ for r in results]},
+              open(out, "w", encoding="utf-8"), ensure_ascii=False, indent=2)
+    print(f"\n报告已落:{out}")
+    return 0 if s["passed"] == s["total"] else 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

+ 82 - 0
tests/test_agent_eval.py

@@ -0,0 +1,82 @@
+"""agent_eval 框架单测:用 mock 跑/收产物/judge,验证行为回归评估逻辑(可进 CI)。
+
+真实 LLM 评估由 evals/run_evals.py 手动/nightly 触发(花钱、要 key),不在默认门禁。
+"""
+from __future__ import annotations
+
+from agentpaas.engine.agent_eval import EvalTask, run_eval_task, summarize
+from agentpaas.engine.pipeline import JudgeResult
+
+
+def _judge(score):
+    return lambda req: JudgeResult(score=score, passed=score >= req.threshold,
+                                   reasons=["mock"], raw="", usage={})
+
+
+def _run(output="done", steps=10, cost=0.1, ws="/ws"):
+    return lambda task: {"output": output, "steps": steps, "cost_usd": cost, "workspace_path": ws}
+
+
+def test_all_good_passes():
+    task = EvalTask(id="t1", input="x", must_produce=["report.md"], rubric="好不好", threshold=0.6)
+    r = run_eval_task(task, run_fn=_run(),
+                      collect_artifacts_fn=lambda ws: {"report.md": "完整内容"},
+                      judge_fn=_judge(0.8))
+    assert r.files_ok and r.judge_passed and r.passed
+    assert r.judge_score == 0.8 and r.steps == 10
+
+
+def test_missing_file_fails():
+    task = EvalTask(id="t2", must_produce=["paper.pdf", "report.md"], rubric="", threshold=0.6)
+    r = run_eval_task(task, run_fn=_run(),
+                      collect_artifacts_fn=lambda ws: {"report.md": "x"},
+                      judge_fn=_judge(1.0))
+    assert not r.files_ok
+    assert r.missing_files == ["paper.pdf"]
+    assert not r.passed
+
+
+def test_judge_below_threshold_fails():
+    task = EvalTask(id="t3", must_produce=[], rubric="质量", threshold=0.7)
+    r = run_eval_task(task, run_fn=_run(),
+                      collect_artifacts_fn=lambda ws: {"a.md": "敷衍"},
+                      judge_fn=_judge(0.4))
+    assert not r.judge_passed and not r.passed
+
+
+def test_error_output_fails():
+    task = EvalTask(id="t4", rubric="", must_produce=[])
+    r = run_eval_task(task, run_fn=_run(output="[DASHSCOPE_ERROR] boom"),
+                      collect_artifacts_fn=lambda ws: {},
+                      judge_fn=_judge(1.0))
+    assert r.error and not r.passed
+
+
+def test_run_exception_captured():
+    def boom(task):
+        raise RuntimeError("run crashed")
+    task = EvalTask(id="t5")
+    r = run_eval_task(task, run_fn=boom,
+                      collect_artifacts_fn=lambda ws: {}, judge_fn=_judge(1.0))
+    assert "run crashed" in r.error and not r.passed
+
+
+def test_no_rubric_skips_judge():
+    task = EvalTask(id="t6", must_produce=["x.md"], rubric="")
+    r = run_eval_task(task, run_fn=_run(),
+                      collect_artifacts_fn=lambda ws: {"x.md": "ok"},
+                      judge_fn=lambda req: (_ for _ in ()).throw(AssertionError("不该调 judge")))
+    assert r.judge_passed and r.passed  # 无 rubric → 不调 judge,judge_passed 默认 True
+
+
+def test_summarize():
+    task = EvalTask(id="a", must_produce=["r.md"], rubric="", )
+    good = run_eval_task(task, run_fn=_run(steps=5, cost=0.2),
+                         collect_artifacts_fn=lambda ws: {"r.md": "x"}, judge_fn=_judge(1))
+    bad = run_eval_task(EvalTask(id="b", must_produce=["miss.pdf"]),
+                        run_fn=_run(steps=9, cost=0.3),
+                        collect_artifacts_fn=lambda ws: {}, judge_fn=_judge(1))
+    s = summarize([good, bad])
+    assert s["total"] == 2 and s["passed"] == 1 and s["pass_rate"] == 0.5
+    assert s["failures"] == ["b"]
+    assert s["total_cost_usd"] == 0.5