將追蹤與黃金基準比對 — Altifigence Documentation

從已知良好的追蹤建立golden-diff基準，以文字或JSON檢查後續追蹤是否符合該基準，並在變更為預期時更新基準。

在Linux上使用Altifigence CLI 0.1.9，從已知良好的.altrace追蹤記錄基準，並檢查後續追蹤是否符合該基準。當每個步驟都在容許範圍內時，檢查會以0結束；任何步驟偏移時則以1結束。

準備兩個追蹤

golden-diff會讀取標頭宣告"encoding":"bincode"的.altrace追蹤。下方的指令碼會寫出run.altrace（已知良好、含一個核心與五個事件的追蹤）與changed.altrace（相同追蹤，但多一個MAC_COMPUTE事件）。每個事件行的格式為core start duration type [api_name]。

PROJECT="$(cd "$HOME/alti-projects/demo-project" && pwd -P)"
cd "$PROJECT"

le() { local n=$1 v=$2; while [ "$n" -gt 0 ]; do printf "\\x$(printf %02x $((v & 255)))"; v=$((v >> 8)); n=$((n - 1)); done; }
str() { le 8 "${#1}"; printf '%s' "$1"; }
write_trace() {
  local out="$1" payload line core start dur type api total=0 header
  local -a lines
  mapfile -t lines
  payload="$(mktemp)"
  {
    for line in "${lines[@]}"; do
      read -r core start dur type api <<<"$line"
      (( start + dur > total )) && total=$((start + dur))
    done
    le 8 "$total"
    le 8 "${#lines[@]}"
    for line in "${lines[@]}"; do
      read -r core start dur type api <<<"$line"
      le 4 "$core"; le 8 "$start"; le 8 "$dur"; str "$type"
      if [ -n "$api" ]; then printf '\x01'; str "$api"; else printf '\x00'; fi
    done
  } >"$payload"
  header="{\"altifigence_trace_version\":\"1.0\",\"trace\":{\"cycles\":$total,\"cores\":1,\"clock_mhz\":1000},\"payload\":{\"encoding\":\"bincode\",\"byte_length\":$(wc -c <"$payload")}}"
  { printf 'ALTR\x01\x01\x00\x00'; le 8 "${#header}"; printf '%s' "$header"; cat "$payload"; } >"$out"
  rm -f "$payload"
}

write_trace run.altrace <<'EVENTS'
0 0 100 MAC_COMPUTE
0 100 50 DMA_READ
0 150 5 API_CALL step_1
0 160 100 MAC_COMPUTE
0 260 100 MAC_COMPUTE
EVENTS

write_trace changed.altrace <<'EVENTS'
0 0 100 MAC_COMPUTE
0 100 50 DMA_READ
0 150 5 API_CALL step_1
0 160 100 MAC_COMPUTE
0 260 100 MAC_COMPUTE
0 360 100 MAC_COMPUTE
EVENTS

wc -c < run.altrace
wc -c < changed.altrace

請確認兩個wc命令都會輸出檔案大小。

376
416

建立基準

alti verification golden-diff emit-profile會讀取一個追蹤並將基準寫入stdout，因此請將其重新導向到檔案。每個API_CALL開始一個步驟，第一個API_CALL之前的事件屬於步驟0。--tolerance-pct會將容許的偏移百分比標記到每一列，預設值為10。

alti verification golden-diff emit-profile --project "$PROJECT" run.altrace > "$PROJECT/reference.jsonl"
grep -v '^#' "$PROJECT/reference.jsonl"

請確認結束狀態為0。檔案以兩行#註解行開頭，之後每個步驟有一列JSON。

{"step":0,"api_name":null,"expect_mac":1,"expect_dma_read":1,"expect_dma_write":0,"expect_barrier":0,"cycle_budget":165,"tolerance_pct":10.0,"abs_tolerance":null}
{"step":1,"api_name":"step_1","expect_mac":2,"expect_dma_read":0,"expect_dma_write":0,"expect_barrier":0,"cycle_budget":232,"tolerance_pct":10.0,"abs_tolerance":null}

每一列會以expect_mac、expect_dma_read、expect_dma_write與expect_barrier儲存觀察到的事件計數。cycle_budget是該步驟涵蓋的週期數，依tolerance_pct所指定的百分比增加後無條件進位：步驟0涵蓋150個週期，步驟1涵蓋210個週期。讀取檔案時會略過空白行與以#開頭的行。

依基準檢查追蹤

alti verification golden-diff check先接受基準，再接受追蹤。文字輸出在由═字元組成的兩行橫幅之間有一個golden-diff:摘要，之後每個步驟一行。下列摘錄省略了橫幅行。

alti verification golden-diff check --project "$PROJECT" reference.jsonl run.altrace; echo "exit=$?"

  golden-diff: all 2 steps within tolerance
step   0 pass
step   1 pass
exit=0

請確認結束狀態為0。請在不使用alti全域--json旗標的情況下執行本節的命令，讓stdout包含報告本身。--format json會輸出報告，--format json-v1會輸出含來源紀錄的版本化結果。兩者都使用與文字格式相同的結束狀態。

alti verification golden-diff check --project "$PROJECT" reference.jsonl run.altrace --format json \
  | jq '{summary, step_count, pass_count, total_metrics, exact_matches, tolerance_passes, metric_mismatches, unexpected_observed_steps}'

{
  "summary": "golden-diff: all 2 steps within tolerance",
  "step_count": 2,
  "pass_count": 2,
  "total_metrics": 10,
  "exact_matches": 8,
  "tolerance_passes": 2,
  "metric_mismatches": 0,
  "unexpected_observed_steps": 0
}

alti verification golden-diff check --project "$PROJECT" reference.jsonl run.altrace --format json-v1 \
  | jq '{schema, operation_id, artifact_schema: .artifact.artifact_schema, engine: .provenance.engine}'

{
  "schema": "altifigence.golden-diff.operation-result.v1",
  "operation_id": "altifigence.verification.golden-diff.v1",
  "artifact_schema": "altifigence.golden-diff.report.v1",
  "engine": "alti-verification"
}

讀取失敗的檢查

changed.altrace的步驟1包含三個MAC_COMPUTE事件，涵蓋310個週期，而基準預期兩個事件與232個週期的預算。

alti verification golden-diff check --project "$PROJECT" reference.jsonl changed.altrace; echo "exit=$?"

  golden-diff: 1 / 2 steps PASS — 1 regressed
step   0 pass
step   1 drift on [mac, cycles]
    └─ mac       observed=3 · expected=2 · tol ±10.0%
    └─ cycles    observed=310 · expected=232 · tol ±10.0%
exit=1

請確認結束狀態為1。alti也會在stderr輸出一行包含exit=1的狀態行。步驟下方只會列出失敗的指標。若要從JSON報告讀取這些指標，請選取pass為false的指標。

alti verification golden-diff check --project "$PROJECT" reference.jsonl changed.altrace --format json \
  | jq -c '.steps[] | select(.is_pass | not) | .metrics[] | select(.pass | not) | {name, observed, expected}'

{"name":"mac","observed":3,"expected":2}
{"name":"cycles","observed":310,"expected":232}

基準列出但追蹤缺少的步驟會輸出step N missing from observed trace。追蹤中沒有基準列的步驟會計入unexpected_observed_steps。兩者都會使檢查以1結束。

處理失敗

當偏移是預期的行為時，請將已變更的追蹤作為新基準，然後再次檢查。

alti verification golden-diff emit-profile --project "$PROJECT" changed.altrace > "$PROJECT/reference.jsonl"
alti verification golden-diff check --project "$PROJECT" reference.jsonl changed.altrace; echo "exit=$?"

  golden-diff: all 2 steps within tolerance
step   0 pass
step   1 pass
exit=0

不是追蹤的檔案會使emit-profile與check以1結束，並輸出下列訊息。請傳入以ALTR開頭的檔案，例如run.altrace。

printf 'this is not a trace\n' > "$PROJECT/notes.txt"
alti verification golden-diff emit-profile --project "$PROJECT" notes.txt
alti verification golden-diff check --project "$PROJECT" reference.jsonl notes.txt

parse error in 'notes.txt': Invalid magic number — expected 'ALTR', got '[116, 104, 105, 115]'
trace parse error for 'notes.txt': invalid .altrace magic

標頭宣告其他酬載編碼的追蹤，會使check以1結束，並輸出此形式的訊息。請傳入標頭宣告"encoding":"bincode"的追蹤，例如run.altrace。

trace parse error for 'other.altrace': unsupported .altrace payload encoding 'flatbuf'

沒有列的基準會使check以1結束，並輸出下列訊息。請再次執行emit-profile，寫出每個步驟一列的基準。

: > "$PROJECT/empty.jsonl"
alti verification golden-diff check --project "$PROJECT" empty.jsonl run.altrace

empty reference file: empty.jsonl

負數或非數值的--tolerance-pct會在引擎啟動前以2結束，錯誤中含有此文字。請傳入0或更大的數字，例如5。

tolerance-pct must be a finite non-negative number

alti verification golden-diff emit-profile --project "$PROJECT" run.altrace --tolerance-pct 5 > "$PROJECT/reference.jsonl"

CLI 引擎指令