将跟踪与黄金基线比较 — Altifigence Documentation

根据已知良好的跟踪创建golden-diff基线，以文本或JSON格式对照该基线检查后续跟踪，并在有意变更时刷新基线。

在Linux上使用Altifigence CLI 0.1.9，根据已知良好的.altrace跟踪记录基线，并对照基线检查后续跟踪。当每个步骤都在容差范围内时，检查以0退出；当任何步骤发生偏差时，以1退出。

准备两个跟踪

golden-diff读取头部声明了"encoding":"bincode"的.altrace跟踪。下面的脚本会写入run.altrace和changed.altrace：前者是包含一个核心和五个事件的已知良好跟踪，后者是在同一跟踪的基础上多一个MAC_COMPUTE事件。每个事件行的格式为core start duration type [api_name]。

PROJECT="$(cd "$HOME/alti-projects/demo-project" && pwd -P)"
cd "$PROJECT"

le() { local n=$1 v=$2; while [ "$n" -gt 0 ]; do printf "\\x$(printf %02x $((v & 255)))"; v=$((v >> 8)); n=$((n - 1)); done; }
str() { le 8 "${#1}"; printf '%s' "$1"; }
write_trace() {
  local out="$1" payload line core start dur type api total=0 header
  local -a lines
  mapfile -t lines
  payload="$(mktemp)"
  {
    for line in "${lines[@]}"; do
      read -r core start dur type api <<<"$line"
      (( start + dur > total )) && total=$((start + dur))
    done
    le 8 "$total"
    le 8 "${#lines[@]}"
    for line in "${lines[@]}"; do
      read -r core start dur type api <<<"$line"
      le 4 "$core"; le 8 "$start"; le 8 "$dur"; str "$type"
      if [ -n "$api" ]; then printf '\x01'; str "$api"; else printf '\x00'; fi
    done
  } >"$payload"
  header="{\"altifigence_trace_version\":\"1.0\",\"trace\":{\"cycles\":$total,\"cores\":1,\"clock_mhz\":1000},\"payload\":{\"encoding\":\"bincode\",\"byte_length\":$(wc -c <"$payload")}}"
  { printf 'ALTR\x01\x01\x00\x00'; le 8 "${#header}"; printf '%s' "$header"; cat "$payload"; } >"$out"
  rm -f "$payload"
}

write_trace run.altrace <<'EVENTS'
0 0 100 MAC_COMPUTE
0 100 50 DMA_READ
0 150 5 API_CALL step_1
0 160 100 MAC_COMPUTE
0 260 100 MAC_COMPUTE
EVENTS

write_trace changed.altrace <<'EVENTS'
0 0 100 MAC_COMPUTE
0 100 50 DMA_READ
0 150 5 API_CALL step_1
0 160 100 MAC_COMPUTE
0 260 100 MAC_COMPUTE
0 360 100 MAC_COMPUTE
EVENTS

wc -c < run.altrace
wc -c < changed.altrace

请确认两条wc命令输出文件大小。

376
416

创建基线

alti verification golden-diff emit-profile读取一个跟踪并将基线写入stdout，因此请将其重定向到文件。每个API_CALL处开始一个步骤，第一个API_CALL之前的事件属于步骤0。--tolerance-pct将允许的偏差百分比标记到每一行上，默认值为10。

alti verification golden-diff emit-profile --project "$PROJECT" run.altrace > "$PROJECT/reference.jsonl"
grep -v '^#' "$PROJECT/reference.jsonl"

请确认退出状态为0。文件以两行#注释开头，然后每个步骤对应一行JSON。

{"step":0,"api_name":null,"expect_mac":1,"expect_dma_read":1,"expect_dma_write":0,"expect_barrier":0,"cycle_budget":165,"tolerance_pct":10.0,"abs_tolerance":null}
{"step":1,"api_name":"step_1","expect_mac":2,"expect_dma_read":0,"expect_dma_write":0,"expect_barrier":0,"cycle_budget":232,"tolerance_pct":10.0,"abs_tolerance":null}

每一行将观察到的事件计数存储为expect_mac、expect_dma_read、expect_dma_write和expect_barrier。cycle_budget是该步骤所跨越的周期数，按tolerance_pct上调并向上取整：步骤0跨越150个周期，步骤1跨越210个周期。读取文件时会跳过空行和以#开头的行。

对照基线检查跟踪

alti verification golden-diff check先接收基线，再接收跟踪。文本输出在由═字符组成的两行横幅之间有一个golden-diff:摘要，然后每个步骤一行。下面的摘录省略了横幅行。

alti verification golden-diff check --project "$PROJECT" reference.jsonl run.altrace; echo "exit=$?"

  golden-diff: all 2 steps within tolerance
step   0 pass
step   1 pass
exit=0

请确认退出状态为0。请在不使用alti全局--json标志的情况下运行本节中的命令，以便stdout中是报告本身。--format json输出报告，--format json-v1输出带有来源信息的版本化结果。两者的退出状态与文本格式相同。

alti verification golden-diff check --project "$PROJECT" reference.jsonl run.altrace --format json \
  | jq '{summary, step_count, pass_count, total_metrics, exact_matches, tolerance_passes, metric_mismatches, unexpected_observed_steps}'

{
  "summary": "golden-diff: all 2 steps within tolerance",
  "step_count": 2,
  "pass_count": 2,
  "total_metrics": 10,
  "exact_matches": 8,
  "tolerance_passes": 2,
  "metric_mismatches": 0,
  "unexpected_observed_steps": 0
}

alti verification golden-diff check --project "$PROJECT" reference.jsonl run.altrace --format json-v1 \
  | jq '{schema, operation_id, artifact_schema: .artifact.artifact_schema, engine: .provenance.engine}'

{
  "schema": "altifigence.golden-diff.operation-result.v1",
  "operation_id": "altifigence.verification.golden-diff.v1",
  "artifact_schema": "altifigence.golden-diff.report.v1",
  "engine": "alti-verification"
}

读取失败的检查

changed.altrace的步骤1包含三个MAC_COMPUTE事件并跨越310个周期，而基线预期两个事件和232个周期的预算。

alti verification golden-diff check --project "$PROJECT" reference.jsonl changed.altrace; echo "exit=$?"

  golden-diff: 1 / 2 steps PASS — 1 regressed
step   0 pass
step   1 drift on [mac, cycles]
    └─ mac       observed=3 · expected=2 · tol ±10.0%
    └─ cycles    observed=310 · expected=232 · tol ±10.0%
exit=1

请确认退出状态为1。alti还会在stderr上输出一行包含exit=1的状态行。步骤下只列出失败的指标。要从JSON报告中读取它们，请选择pass为false的指标。

alti verification golden-diff check --project "$PROJECT" reference.jsonl changed.altrace --format json \
  | jq -c '.steps[] | select(.is_pass | not) | .metrics[] | select(.pass | not) | {name, observed, expected}'

{"name":"mac","observed":3,"expected":2}
{"name":"cycles","observed":310,"expected":232}

基线中列出但跟踪中缺少的步骤会输出step N missing from observed trace。跟踪中没有基线行的步骤会计入unexpected_observed_steps。两种情况都会使检查以1退出。

处理失败

当偏差是预期行为时，请将变更后的跟踪作为新基线，然后再次检查。

alti verification golden-diff emit-profile --project "$PROJECT" changed.altrace > "$PROJECT/reference.jsonl"
alti verification golden-diff check --project "$PROJECT" reference.jsonl changed.altrace; echo "exit=$?"

  golden-diff: all 2 steps within tolerance
step   0 pass
step   1 pass
exit=0

不是跟踪的文件会使emit-profile和check以1退出，并输出下面的消息。请传入以ALTR开头的文件，例如run.altrace。

printf 'this is not a trace\n' > "$PROJECT/notes.txt"
alti verification golden-diff emit-profile --project "$PROJECT" notes.txt
alti verification golden-diff check --project "$PROJECT" reference.jsonl notes.txt

parse error in 'notes.txt': Invalid magic number — expected 'ALTR', got '[116, 104, 105, 115]'
trace parse error for 'notes.txt': invalid .altrace magic

头部声明了其他载荷编码的跟踪会使check以1退出，并输出这种形式的消息。请传入头部声明了"encoding":"bincode"的跟踪，例如run.altrace。

trace parse error for 'other.altrace': unsupported .altrace payload encoding 'flatbuf'

没有行的基线会使check以1退出，并输出下面的消息。请重新运行emit-profile，写入每个步骤一行的基线。

: > "$PROJECT/empty.jsonl"
alti verification golden-diff check --project "$PROJECT" empty.jsonl run.altrace

empty reference file: empty.jsonl

负数或非数字的--tolerance-pct会在引擎启动之前以2退出，错误中包含这段文本。请传入0或更大的数，例如5。

tolerance-pct must be a finite non-negative number

alti verification golden-diff emit-profile --project "$PROJECT" run.altrace --tolerance-pct 5 > "$PROJECT/reference.jsonl"

CLI 引擎命令