Announcing Phisch

A few weeks ago, we released Phisch, our first language model architecture, developed entirely in Rust using the Burn ML framework. At the time, there was no way to verify that it worked, since no pre-trained models were available to load and test, so we worked quietly until we had something ready.

Today, we are pleased to explain what Phisch is and, at the same time, release the first fully trained Phisch model: Poetry Phisch.

Why create Phisch?

Crab Lab emerged from two converging issues with current AI models: they are too large for most users’ hardware, and they are closed-source, meaning no one outside the companies that develop them can train one of these models independently. Even open-weight models have begun to grow in size to compete with state-of-the-art closed-source alternatives. This is what inspired us to create and release our own model.

In creating Phisch, we focused on two things: ensuring that all tools built around the model were well-documented and easy for any machine learning enthusiast to follow, and making it simple to adjust hyperparameters to fit each user’s hardware.

What is Phisch?

Phisch is a language model for text generation. It is based on a decoder-only Transformer architecture, modernized with improvements introduced by Llama (Touvron et al., 2023): RMSNorm, RoPE, and SwiGLU.

We chose Rust because of the tools and capabilities the language offers. Thanks to these features, compiling Phisch across various environments and systems is straightforward, and a single binary handles training, text generation, model serving, and exporting models in safetensors format.

Built on Burn, Phisch uses its wgpu backend, enabling execution on any GPU compatible with Vulkan, Metal, or DirectX 12, without relying on CUDA or any specific hardware manufacturer. This, combined with configurable hyperparameters such as the number of layers, embedding dimensions, attention heads, and context length, allows the model to be adapted to the available resources, whether a dedicated or an integrated GPU.

Introducing Poetry Phisch

To demonstrate Phisch’s capabilities, we trained Poetry Phisch, a 3-million-parameter model, on a custom dataset of 2,230 classic English poetry books from Project Gutenberg. We only included books that are in the public domain in the US and whose credited contributors all died at least 70 years ago. In total, we collected approximately 87 million tokens: about 83 million for training and 4.4 million for validation.

Training was performed on a consumer-grade GPU (an NVIDIA GTX 1660 with 6 GB of VRAM) and took about four and a half hours, split into two runs of three epochs each. In the first run, training loss stabilized around 3.49 by the end of the third epoch. The second run resumed from the first run’s best checkpoint with a lower learning rate to confirm convergence, and validation loss improved only negligibly, ending at 3.69.

The result is a model that mimics the structure and style of its training texts, although it does not yet manage to produce meaningful sentences or maintain a coherent idea across multiple lines. In our tests, we also found no instances of the model reproducing verbatim excerpts from the dataset. Some examples:

Prompt: The bird

The bird is singing, and the stars are white.
And I am sick of all things long ago--
I cannot tell you why I love you.

Prompt: Love

Love is a bird of the moon.
And I have heard my voice, and can say:
'The world is in the night,' we said.
He looked at me from my eyes.

Getting started

To try Poetry Phisch, download it from Hugging Face and follow the instructions in its model card.

To install Phisch and train your own model, check the README, which walks through a simplified training pipeline. For a detailed explanation of every command and flag, see USAGE.md.

What’s next?

You can try Poetry Phisch right now in our playground, where future Phisch models will also be available. And if you train your own model, we’d love to hear about it! Feel free to open an issue on GitHub with feedback, questions, or results.