tiktoken

# ⏳ tiktoken

tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with OpenAI's models.

```python import tiktoken enc = tiktoken.get_encoding("o200k_base") assert enc.decode(enc.encode("hello world")) == "hello world"

# To get the tokeniser corresponding to a specific model in the OpenAI API: enc = tiktoken.encoding_for_model("gpt-4o") ```

The open source version of `tiktoken` can be installed from [PyPI](https://pypi.org/project/tiktoken): ``` pip ins