Tokenize and detokenize text with a local model's tokenizer

Usage

llm_tokenize(config, text, special = FALSE)

llm_detokenize(config, ids)

Arguments

config: An llm_config() for provider "llamacpp".
text: One string.
special: If TRUE, special-token text such as <|im_start|> is read as the special token; otherwise as plain text.
ids: Integer token ids.

Value

llm_tokenize() returns a tibble with id, piece (the token's text), and bytes (its raw bytes; a token can be part of a multi-byte character, whose piece alone is then not valid text). llm_detokenize() returns one string.

Examples

if (llm_local_ready("smollm2-135m")) {
  cfg <- llm_config("llamacpp", "smollm2-135m")
  llm_local_start("smollm2-135m", threads = 2)
  tk <- llm_tokenize(cfg, "The capital of France is Paris.")
  llm_detokenize(cfg, tk$id)
  llm_local_stop()
}