Skip to content(if available)orjump to list(if available)

Calculate the number of language model tokens for a string

Calculate the number of language model tokens for a string

2 comments

·February 2, 2025
Visit

sva_

  import tiktoken
  enc = tiktoken.encoding_for_model('o1-mini')
  len(enc.encode('some string'))

NitpickLawyer

Does that work on anything other than openai?