Token Accounting

Token Accounting for Claude / Codex / ZCode / Qwen

This document explains how Open ACE collects token usage for the four local tools claude, codex, zcode, and qwen, how it computes daily and message-level metrics, how those metrics are stored, and which downstream services/pages consume which table.

Target readers:

users who want to understand why Open ACE numbers can differ from provider dashboards
maintainers who need to debug or change fetchers
contributors adding a new local tool or adjusting usage semantics

Concepts

This opening section replaces the former CONCEPTS.md and restates its four core concepts against the actual code model.

1. Request (API call)

An independent model invocation attributed to one billing unit. Open ACE's request_count is tool-specific:

Claude / Qwen: counted per logical assistant message, deduplicated by stable message id when available; a zero-token assistant message can still count as a request
Codex: counted per task_started event (one per reconstructed turn), not per assistant row
ZCode: attributed per turn_usage row (the authoritative billing record)

See section 3 for the per-tool pipelines.

2. Message

Every persisted row in daily_messages / session_messages carries a normalized role from the canonical set:

Role	Meaning
`user`	Messages sent by the user
`assistant`	AI-generated responses
`system`	System/context messages
`tool`	Tool execution results

The canonical values are defined in app/utils/roles.py (normalize_message_role), and every write path into daily_messages funnels through it at the write boundary (app/repositories/message_repo.py documents the accepted roles as user, assistant, system, tool). Variant tool-result spellings (toolResult, tool_result) collapse to tool; blank/None input becomes unknown. There is no separate message_user / message_toolresult / message_error taxonomy in the code.

Example:

user: "Check the weather"           → role=user      +1
assistant: "Let me check..."        → role=assistant +1
tool: {weather data}                → role=tool      +1
assistant: "It's sunny today..."    → role=assistant +1

3. Session

A tool-level session — the process-level unit identified by agent_session_id. Sessions belong to a tool: claude, qwen, codex, zcode, and openclaw each have their own fetcher (scripts/fetch_*.py) and session layout. One session contains multiple conversations and may span multiple local calendar days (which is why daily grouping follows turn/message timestamps, not session creation time).

4. Conversation

One round of dialogue: from a user sending a message to the AI completing the final response, including any tool calls in between. Stored as conversation_id ("one round of conversation" per the save_message docstring in app/repositories/message_repo.py).

Concept relationship

Session (tool process: claude / qwen / codex / zcode / openclaw)
├── Conversation 1 (one round)
│   ├── user message          (role=user)
│   ├── assistant response    (role=assistant)
│   ├── tool result           (role=tool)
│   └── final assistant reply (role=assistant)
├── Conversation 2
│   ├── user message          (role=user)
│   └── assistant response    (role=assistant)
└── Conversation N

Statistics example

Conversation 1:
  user: "Check the weather"
  │
  ├── Request 1 ──> assistant: "Let me check..."   (role=assistant)
  │
  ├── Request 2 ──> tool: {weather data}            (role=tool)
  │
  └── Request 3 ──> assistant: "It's sunny..."      (role=assistant)

Conversation 2:
  user: "What about tomorrow?"
  │
  └── Request 4 ──> assistant: "Tomorrow..."        (role=assistant)

Results:

Requests: 4 model invocations (how they map to request_count depends on the tool — see the Request concept above)
Messages: 6 rows (2 user + 3 assistant + 1 tool)
Conversations: 2 rounds

Database fields

Field	Description
`agent_session_id`	Tool session identifier (process level)
`conversation_id`	Conversation identifier (one round of dialogue)
`feishu_conversation_id`	Feishu conversation identifier

1. End-to-end flow

local JSONL / SQLite
  -> scripts/fetch_*.py
  -> scripts/shared/db.py
     -> daily_usage
     -> daily_messages
     -> agent_sessions
     -> session_messages
     -> daily_stats / hourly_stats
     -> user_daily_stats
  -> app/repositories/* / app/services/*
  -> Work / Manage pages, quota, reporting, analytics

At a high level:

daily_usage is the day/tool/host aggregate fact table
daily_messages is the message-level analytics fact table
agent_sessions / session_messages power workspace/session views
daily_stats, hourly_stats, and user_daily_stats are derived aggregates

2. Shared semantics

2.1 Core fields

Field	Meaning	Main tables
`tokens_used`	Open ACE's total-token value for that record	`daily_usage`, `daily_messages`, `agent_sessions`, `session_messages`
`input_tokens`	non-cached input tokens for that record	`daily_usage`, `daily_messages`, `agent_sessions`, `session_messages`
`output_tokens`	output tokens	`daily_usage`, `daily_messages`, `agent_sessions`, `session_messages`
`cache_tokens`	total cache tokens	only stored separately in `daily_usage`
`request_count`	Open ACE request-count semantic	`daily_usage`, `agent_sessions`, `user_daily_stats`

Important caveats:

daily_messages does not have its own cache_tokens column.
Cache is preserved separately only in daily_usage.cache_tokens.
If a consumer already reads tokens_used, it usually must not add cache_tokens again.

2.2 Intended invariant

For these four tools, the current intent is to keep the stored totals as close as possible to:

tokens_used == non-cached input_tokens + output_tokens + cache_tokens

But provider-native semantics differ:

Claude exposes cache separately, so Open ACE explicitly adds cache into tokens_used
Codex provider total_tokens already includes cached input
Qwen provider promptTokenCount includes cache, while totalTokenCount is preserved as the provider total
Zcode uses turn_usage as the source of truth

3. Tool-by-tool collection and computation

3.1 Claude

Source

path: ~/.claude/projects/**.jsonl
script: scripts/fetch_claude.py

How usage is extracted

Claude local logs are JSONL
usage is read from entry["usage"] or entry["message"]["usage"]
key functions:
- extract_tokens_from_entry()
- process_jsonl_file()
- _merge_messages_by_id()

Field mapping

input_tokens comes from input_tokens
output_tokens comes from output_tokens
cache_read_tokens comes from cache_read_input_tokens
cache_creation_tokens comes from cache_creation_input_tokens
tokens_used is computed as input + output + cache_read + cache_creation

Why message-id merge exists

one logical Claude message can span multiple JSONL lines
line-by-line inserts would duplicate token accounting and lose structured content
Open ACE merges the logical message first, then writes daily_messages / session_messages

request_count

counted per logical assistant message
deduplicated by stable message id when available
a zero-token assistant message can still count as a request

3.2 Codex

Source

path: ~/.codex/sessions/YYYY/MM/DD/rollout-*.jsonl
script: scripts/fetch_codex.py

Key event types

task_started
turn_context
response_item
token_count
task_complete

How usage is reconstructed

Codex does not store billing usage as one field per final message
Open ACE rebuilds turns from the event stream
task_started opens a turn
response_item collects user / assistant messages
token_count accumulates turn usage
task_complete closes the turn

Why attribution is per turn, not per session

a Codex session can span many hours or multiple days
putting all tokens on the first assistant row distorts daily and hourly stats
current logic reconstructs turns and attributes each turn to the initiating user message

Field mapping

tokens_used comes from accumulated last_token_usage.total_tokens
cache_tokens comes from cached_input_tokens
input_tokens is computed as input_tokens - cached_input_tokens
output_tokens comes from output_tokens
thoughts_tokens only participates in daily aggregation and is not stored separately in daily_messages

Important semantic note

for current local Codex logs, token_count.last_token_usage behaves as event-level increments and should be summed
cached_input_tokens is already part of provider total, not an extra add-on

request_count

counted per task_started
not per assistant message row

Why re-import deletes old rows first

parser fixes can move token attribution from assistant rows to user rows
delete_messages_for_agent_sessions() clears old rows by tool_name + host_name + agent_session_id
then the authoritative snapshot is re-written

3.3 Zcode

Source

path: ~/.zcode/cli/db/db.sqlite
script: scripts/fetch_zcode.py
source tables: session, message, part, turn_usage

Why it differs from the other three

Zcode stores source data relationally in SQLite, not JSONL
message / part are useful for transcript reconstruction
turn_usage is the authoritative source for token accounting

How usage is computed

ZcodeSession from remote-agent/session_sync.py is reused for transcript/project parsing
token attribution is then done by querying turn_usage
key functions:
- _get_turn_usage_rows()
- _get_turn_usage_by_date()
- process_zcode_session()

Field mapping

tokens_used comes from computed_total_tokens
input_tokens comes from turn_usage.input_tokens
output_tokens comes from turn_usage.output_tokens
cache_tokens is computed as cache_creation_input_tokens + cache_read_input_tokens

Why date grouping uses turn_usage.started_at

one session may span multiple local calendar days
the authoritative billing timestamp for Zcode is the turn start
so daily_usage is split by turn date, not by session creation time or dominant message day

Message-level attribution

each turn is attached to the user message referenced by turn_usage.user_message_id
this keeps daily_messages / hourly_stats aligned with real turn timing

What happens on mismatch

partial match: warning is printed
- daily_usage remains authoritative
- daily_messages / agent_sessions may be incomplete for unmatched turns
full mismatch: fallback warning is printed and the whole session total is injected into the first assistant message

3.4 Qwen

Source

path: ~/.qwen/projects/**/chats/*.jsonl
some older layouts also use direct *.jsonl
script: scripts/fetch_qwen.py

How usage is extracted

usage comes from usageMetadata
key functions:
- extract_tokens_from_entry()
- process_jsonl_file()

Field mapping

prompt_tokens comes from promptTokenCount
candidates_tokens comes from candidatesTokenCount
thoughts_tokens comes from thoughtsTokenCount
cached_tokens comes from cachedContentTokenCount
tokens_used comes from totalTokenCount

Important semantic note

promptTokenCount includes cached context
Open ACE computes actual_input_tokens = promptTokenCount - cachedContentTokenCount
daily_messages.input_tokens stores actual_input_tokens
tokens_used still preserves provider totalTokenCount

Why thoughts are not added again

thoughtsTokenCount is treated as an extra observation dimension, not something that should automatically be added on top of provider total
otherwise Open ACE would inflate totals beyond provider semantics

request_count

counted per assistant message
deduplicated by message id when available

4. How data is stored

4.1 `daily_usage`

Role

day/tool/host aggregate fact table
suitable for trend charts, totals, quota, ROI, and cost estimation

Write path

save_usage() in scripts/shared/db.py

Main columns

date
tool_name
host_name
tokens_used
input_tokens
output_tokens
cache_tokens
request_count
models_used

4.2 `daily_messages`

Role

message-level analytics fact table
suitable for hourly analysis, timelines, sender attribution, and conversation/project dimensions

Write path

save_messages_batch() in scripts/shared/db.py

Important constraint

there is no separate cache_tokens column here
this is an analytics fact table, not the workspace runtime transcript authority
direct SUM(tokens_used) requires understanding the tool-specific attribution model

4.3 `agent_sessions`

Role

session summary table
used for workspace session lists and session detail headers

Update path

each fetcher has an update_agent_sessions_stats() implementation

Typical fields

message_count
total_tokens
request_count
model
project_path
updated_at

4.4 `session_messages`

Role

transcript mirror for session detail pages
allows fetchers to populate session replay data directly

Update path

usually inserted from the fetcher's update_agent_sessions_stats()

4.5 `daily_stats` / `hourly_stats`

Role

derived aggregate tables built from daily_messages

Refresh behavior

after save_messages_batch() succeeds, _refresh_daily_stats_for_messages(messages) rebuilds:
- daily_stats
- hourly_stats

4.6 `user_daily_stats`

Role

per-user daily aggregate table
used for quota checks, trends, and fast user-level reads

Refresh behavior

after save_messages_batch() finishes, scripts/shared/user_stats_helper.py
calls app/services/user_stats_aggregator.py
which aggregates daily_messages and agent_sessions into user_daily_stats

5. How downstream code consumes these tables

Module	Primary source	Why
`app/repositories/usage_repo.py`	`daily_usage` first	totals, per-tool summaries, CSV, request counts; avoids `daily_messages` JOIN multiplication
`app/repositories/message_repo.py`	`daily_messages`	hourly usage, message timeline, sender/project/conversation analysis
`app/services/analysis_service.py`	`message_repo` + `hourly_stats`	trend and hourly views combine raw message aggregations with derived tables
`app/services/user_stats_aggregator.py`	`daily_messages` + `agent_sessions`	builds per-user daily aggregates
Work session detail	`agent_sessions` + `session_messages`	does not rely on `daily_messages` as the normal runtime source
Manage usage / analysis	`daily_usage`, `daily_messages`, `daily_stats`, `hourly_stats`	depends on whether the page needs totals, trends, or detail

Practical rules:

For "how many tokens were used that day", prefer daily_usage
For "what happened in this hour", use daily_messages or hourly_stats
For "show me this session transcript", use session_messages
For "how much has this user used today", prefer user_daily_stats

6. Common pitfalls

6.1 Why can Open ACE differ from provider dashboards?

Common reasons:

provider dashboards and local logs refresh at different times
provider-side buckets may exist even when local logs do not expose them
attribution may be session-based, turn-based, or message-based
provider-specific total/cache/thought semantics differ

Differences do not automatically mean a bug. First identify which layer differs:

missing source logs
fetcher attribution
comparing daily_usage to daily_messages even though they serve different purposes

6.2 Is cache included in total?

For these four tools, Open ACE currently treats cache as part of the total. The difference is only whether:

the provider already includes cache in its native total, or
Open ACE has to add cache into tokens_used itself

6.3 Why can `daily_usage` differ from `SUM(daily_messages.tokens_used)`?

Because:

daily_usage is the tool-level aggregate authority
daily_messages is the message-level attribution layer
some tools attribute turns to user rows, not assistant rows
fallback or partial-match cases can keep daily_usage complete while message/session attribution is only approximate

6.4 Why can a re-fetch change historical numbers?

Re-fetch does more than re-insert rows. It may include:

message-id merge fixes
turn-attribution fixes
stale assistant-token cleanup
deleting and rebuilding old session rows

If parser logic changes, historical attribution can be corrected too.

6.5 Why should some pages avoid summing `daily_messages.tokens_used` directly?

Because daily_messages is an analytics fact table, not the universal billing fact.

It is ideal for:

timelines
hourly patterns
sender / project / conversation analysis

But for:

daily totals
tool-to-tool comparisons
quota deduction

consumers should prefer daily_usage or user_daily_stats.

7. Maintenance guidance

When adding a new tool or changing a fetcher, check these questions in order:

What is the provider/source semantic for usage fields?
Does tokens_used already include cache or thoughts?
Should attribution be per session, per turn, or per message?
Are daily_usage and daily_messages being kept in their intended roles?
Does a parser change require stale session-row cleanup?
Are there fetcher unit tests or targeted regression tests covering the semantics?

Useful code entry points:

scripts/fetch_claude.py
scripts/fetch_codex.py
scripts/fetch_zcode.py
scripts/fetch_qwen.py
scripts/shared/db.py
scripts/shared/user_stats_helper.py
app/repositories/usage_repo.py
app/repositories/message_repo.py
app/services/analysis_service.py
app/services/user_stats_aggregator.py

Token Accounting for Claude / Codex / ZCode / Qwen

Concepts​

1. Request (API call)​

2. Message​

3. Session​

4. Conversation​

Concept relationship​

Statistics example​

Database fields​

1. End-to-end flow​

2. Shared semantics​

2.1 Core fields​

2.2 Intended invariant​

3. Tool-by-tool collection and computation​

3.1 Claude​

3.2 Codex​

3.3 Zcode​

3.4 Qwen​

4. How data is stored​

4.1 daily_usage​

4.2 daily_messages​

4.3 agent_sessions​

4.4 session_messages​

4.5 daily_stats / hourly_stats​

4.6 user_daily_stats​

5. How downstream code consumes these tables​

6. Common pitfalls​

6.1 Why can Open ACE differ from provider dashboards?​

6.2 Is cache included in total?​

6.3 Why can daily_usage differ from SUM(daily_messages.tokens_used)?​

6.4 Why can a re-fetch change historical numbers?​

6.5 Why should some pages avoid summing daily_messages.tokens_used directly?​

7. Maintenance guidance​

Concepts

1. Request (API call)

2. Message

3. Session

4. Conversation

Concept relationship

Statistics example

Database fields

1. End-to-end flow

2. Shared semantics

2.1 Core fields

2.2 Intended invariant

3. Tool-by-tool collection and computation

3.1 Claude

3.2 Codex

3.3 Zcode

3.4 Qwen

4. How data is stored

4.1 `daily_usage`

4.2 `daily_messages`

4.3 `agent_sessions`

4.4 `session_messages`

4.5 `daily_stats` / `hourly_stats`

4.6 `user_daily_stats`

5. How downstream code consumes these tables

6. Common pitfalls

6.1 Why can Open ACE differ from provider dashboards?

6.2 Is cache included in total?

6.3 Why can `daily_usage` differ from `SUM(daily_messages.tokens_used)`?

6.4 Why can a re-fetch change historical numbers?

6.5 Why should some pages avoid summing `daily_messages.tokens_used` directly?

7. Maintenance guidance