Posts on anotherjesse.com

Experimenting With Transformers & Nokens

Mon, 04 Mar 2024 00:00:00 +0000

Sometimes a single word isn’t enough. What if you could blend tokens or use non-tokens (aka nokens¹)?

Let’s explore these ideas, learn more about transformers: the architecture and Hugging Face library.

Installing & Setup

I’m going to use Microsoft’s Phi-2 as it is smaller than Mistral / Llama2. The ideas should transfer to other transformer models.

After pip install transformers torch, we can load our language model to experiment with using the Hugging Face library.

Proteus

Sun, 03 Mar 2024 00:00:00 +0000

Parti Prompt Disclaimer

Many of these prompts are challenging for current generative imagery models. Compare: 253 “a motorcycle” vs 999 “a black background with a large yellow square”.

Drag the slider (or type a number between 0 and 1632) to change the prompt. Additionally, you can click any image to see the full-size verison.

Playground V2.5

Tue, 27 Feb 2024 00:00:00 +0000

Note

I’m Still rendering Parti Prompt with Playground … this is a preview

Parti Prompt Disclaimer

Many of these prompts are challenging for current generative imagery models. Compare: 253 “a motorcycle” vs 999 “a black background with a large yellow square”.

Drag the slider (or type a number between 0 and 1632) to change the prompt. Additionally, you can click any image to see the full-size verison.

SDXL lightning

Wed, 21 Feb 2024 00:00:00 +0000

Today ByteDance released SDXL Lightning (adding to the family of distilled/faster SDXL versions Turbo, LCM). From the paper, we can improvements in the quality and how close it matches SDXL.

Exploring Parti Prompts

I recently rendered SDXL as part of parti prompts, and Lightning is quick - let’s make it easy to compare.

Below, each row is a different (initial noise) seed, whereas columns are different models (SDXL, Lightning 1 step, …)

Parti Prompts

Mon, 19 Feb 2024 00:00:00 +0000

Stable Cascade

SDXL

SDXL images were generated with 25 steps using the FP16 variant

The Stable Cascade 2 Step

Sun, 18 Feb 2024 00:00:00 +0000

Stabilty AI has shared an early release of Stable Cascade (built upon Würstchen). Whereas Stable Diffusion uses a single UNet followed by a VAE, this model uses 2 UNets - resulting in improvements in inference time and training resources.

Rorschach Test For LLaVA LLM

Sat, 30 Dec 2023 00:00:00 +0000

LLaVA works by teaching an LLM (Llama2) to “understand” embeddings from an image model (CLIP). What happens if you replace the embeddings with randomness and ask the LLM questions about what it sees?

Hacking LLaVA

After loading the LLaVA model, we can inject a function that runs after clip embed but before projection to the LLMs token latent space.

# load LLaVA using HF transformers
tokenizer, model, image_processor, context_len = load_pretrained_model(
 "liuhaotian/llava-v1.5-13b",
 model_name="llava-v1.5-13b",
 model_base=None,
 load_8bit=False,
 load_4bit=False)

# inject hook that allows modifying the generated clip embed
tower = model.get_vision_tower() 
tower.register_forward_hook(random_forward_hook)

# if global alter_seed set, override embed with randn of same shape
alter_seed = None
def random_forward_hook(module, input, output):
 if not alter_seed is None:
 print("overriding image embedding with seed: ", alter_seed)
 torch.manual_seed(alter_seed)
 output[0] = torch.randn_like(output[0])
 return output

You can try it out by running llava-lies on replicate. This model still asks for an image, to simplify the modification of LLaVA’s code. The hook random_forward_hook above replaces the output with normalized seeded noise.

Kandinksy 3

Wed, 22 Nov 2023 00:00:00 +0000

Kandinsky 3 is the latest from the team that brought previous versions of the model.

We conducted a lot of experiments on choosing an architecture and did a lot of work with data to make text understanding and generation quality better, and the architecture itself simpler and more concise. We also made our model more “domestic”: now it is much better oriented in the Russian and Soviet cultural field.

Lazy AI Coding Bots

Sat, 03 Dec 2022 00:00:00 +0000

I’ve added a new command to my telegram bot: js. This command will ask GPT3 to generate javascript code and then run it.

For instance if I say ask it to generate a function to reverse a string, it will create function that reverses the letters one at a time.

As GPT3 tended to respond with code that defined functions, I realized I could build up a library of functions! And by sending the list of functions as context for further runs, further functions could be based on existing functions.

Being Misunderstood

Tue, 05 Apr 2022 00:00:00 +0000

I really enjoyed the David Whyte’s Clear Mind, Wild Heart.

About 2/3 in, he said something that reshaped how I go about exploring the world.

Being ok with being misunderstood at the beginning is important. New ideas need to be protected and nurtured.

when you actually tell others what your secret desires are - in your work, in your life.

a good diagnostic feature that you are doing something unique - something that is true to the complexion your character - is that other people should not understand entirely what you are doing

Programmable Cat

Sat, 09 Oct 2021 00:00:00 +0000

Idea

The family was excited to decorate to halloween this year, and several of the decorations came with Try Me buttons. The buttons were external to the decoration - and seem to be simple momentary switches (when they are pressed the circuit is closed for the duration of the button press).

It would be neat to add these to our programmable space using a tiny ESP-8266 microcontroller / MQTT.

Based on success of connecting Miffy to our local programmable space, this seemed approachable for an electronics novice like me.

Alien Love Worms

Sun, 28 Oct 2018 00:00:00 +0000

Inspiration

Around the time the glowforge was announced, I saw some inspiring art that had many layered cuts. I thought they were paper, turns out they are by an artist named Gabriel Schama, and are:

huge! many are six to eight feet tall/wide
cut in a (thin) plywood!

Presenting “Alien Love Worm”

For my first design I wanted to do a paper cut. Living near the coast, I wanted to encorporate a whale tail, waves, a peace sign and hearts.

Seamless Loops

Sun, 28 Oct 2018 00:00:00 +0000

Using this tutorial in processing, and Golan Levin’s sketch in processing, started to understand how to make seamlessly looping animations. I took Golan’s awesome demo and ported it to p5.js (live demo). This gif made the technique click for me.

The basic idea is think of the random values as locations on a clock face. This gives 1D noise over time (shown in the image above). For 2D you can use 3D noise space and think about a line sweeping through.

Miffy Super Powers

Fri, 23 Dec 2016 00:00:00 +0000

When my oldest daughter was born, my dear friend Britt Selvitelle gave us a gift of Miffy Lamp from Japan.

My problems with power

Miffy had two “problems”, the first being the power cord for the lamp didn’t fit into american outlets. The second was the bulb being incandescent consumed lots of power and created lots of heat (and rather bright if I remember correctly).

Solution: Let’s put it on the internet!

Having learned about ESP-8266, a cheap & tiny microcontroller that has WiFi, I thought it would be cool to give the bunny an API.