stormlog.infer.tokens

Token counting helpers for inference profiling.

Functions

build_token_counter(*, tokenizer, model[, ...])

Build the requested token counter, falling back to estimates in auto mode.

generate_prompt(target_tokens, counter, *, seed)

Generate a deterministic prompt near the requested token count.

Classes

EstimatedTokenCounter()

A deterministic fallback counter based on whitespace-like chunks.

TiktokenCounter(*, model, encoding_name)

Token counter backed by tiktoken.

TokenCount(value, source, exact)

A token count plus provenance.

TokenCounter(*args, **kwargs)

Counter interface used by workload generation and fallback accounting.

TransformersTokenCounter(*, model)

Token counter backed by transformers.AutoTokenizer.
class stormlog.infer.tokens.TokenCount(value, source, exact)[source]

Bases: object

A token count plus provenance.

Parameters:

  • value (int)

  • source (str)

  • exact (bool)

value: int
source: str
exact: bool
class stormlog.infer.tokens.TokenCounter(*args, **kwargs)[source]

Bases: Protocol

Counter interface used by workload generation and fallback accounting.

source: str
exact: bool
count_text(text)[source]

Count tokens in a text payload.

Parameters:

text (str)

Return type:

TokenCount

class stormlog.infer.tokens.EstimatedTokenCounter[source]

Bases: object

A deterministic fallback counter based on whitespace-like chunks.

source = 'estimated'
exact = False
count_text(text)[source]
Parameters:

text (str)

Return type:

TokenCount

class stormlog.infer.tokens.TiktokenCounter(*, model, encoding_name)[source]

Bases: object

Token counter backed by tiktoken.

Parameters:

  • model (str | None)

  • encoding_name (str | None)

source = 'tiktoken'
exact = True
count_text(text)[source]
Parameters:

text (str)

Return type:

TokenCount

class stormlog.infer.tokens.TransformersTokenCounter(*, model)[source]

Bases: object

Token counter backed by transformers.AutoTokenizer.

Parameters:

model (str)

source = 'transformers'
exact = True
count_text(text)[source]
Parameters:

text (str)

Return type:

TokenCount

stormlog.infer.tokens.build_token_counter(*, tokenizer, model, tokenizer_model=None, tiktoken_encoding=None, strict=False)[source]

Build the requested token counter, falling back to estimates in auto mode.

Parameters:

  • tokenizer (str)

  • model (str)

  • tokenizer_model (str | None)

  • tiktoken_encoding (str | None)

  • strict (bool)

Return type:

TokenCounter

stormlog.infer.tokens.generate_prompt(target_tokens, counter, *, seed)[source]

Generate a deterministic prompt near the requested token count.

Parameters:
Return type:

str