Experimenting With Transformers & Nokens

Mon, 04 Mar 2024 00:00:00 +0000

Sometimes a single word isn’t enough. What if you could blend tokens or use non-tokens (aka nokens¹)?

Let’s explore these ideas, learn more about transformers: the architecture and Hugging Face library.

Installing & Setup

I’m going to use Microsoft’s Phi-2 as it is smaller than Mistral / Llama2. The ideas should transfer to other transformer models.

After pip install transformers torch, we can load our language model to experiment with using the Hugging Face library.