A Maker's Exploration of Generative Art, AI, Coding on anotherjesse.com

Phyllotactic FFT Spirals

Thu, 07 Mar 2024 00:00:00 +0000

Modifying phyllotactic patterns using fft of microphone input.

Note: to use the mic, the browser requires a user interaction – Click the rotating circle to begin!

Using the microphone - learning to love the FFT

The FFT (see 3b1b - But what is the Fourier Transform? ) converts a signal from the time domain to the frequency domain…

What?

Time domain means the value the microphone is receiving at each point in time.

Experimenting With Transformers & Nokens

Mon, 04 Mar 2024 00:00:00 +0000

Sometimes a single word isn’t enough. What if you could blend tokens or use non-tokens (aka nokens¹)?

Let’s explore these ideas, learn more about transformers: the architecture and Hugging Face library.

Installing & Setup

I’m going to use Microsoft’s Phi-2 as it is smaller than Mistral / Llama2. The ideas should transfer to other transformer models.

After pip install transformers torch, we can load our language model to experiment with using the Hugging Face library.

Proteus

Sun, 03 Mar 2024 00:00:00 +0000

Parti Prompt Disclaimer

Many of these prompts are challenging for current generative imagery models. Compare: 253 “a motorcycle” vs 999 “a black background with a large yellow square”.

Drag the slider (or type a number between 0 and 1632) to change the prompt. Additionally, you can click any image to see the full-size verison.

Playground V2.5

Tue, 27 Feb 2024 00:00:00 +0000

Note

I’m Still rendering Parti Prompt with Playground … this is a preview

Parti Prompt Disclaimer

Many of these prompts are challenging for current generative imagery models. Compare: 253 “a motorcycle” vs 999 “a black background with a large yellow square”.

Drag the slider (or type a number between 0 and 1632) to change the prompt. Additionally, you can click any image to see the full-size verison.

SDXL lightning

Wed, 21 Feb 2024 00:00:00 +0000

Today ByteDance released SDXL Lightning (adding to the family of distilled/faster SDXL versions Turbo, LCM). From the paper, we can improvements in the quality and how close it matches SDXL.

Exploring Parti Prompts

I recently rendered SDXL as part of parti prompts, and Lightning is quick - let’s make it easy to compare.

Below, each row is a different (initial noise) seed, whereas columns are different models (SDXL, Lightning 1 step, …)

Parti Prompts

Mon, 19 Feb 2024 00:00:00 +0000

Stable Cascade

SDXL

SDXL images were generated with 25 steps using the FP16 variant

The Stable Cascade 2 Step

Sun, 18 Feb 2024 00:00:00 +0000

Stabilty AI has shared an early release of Stable Cascade (built upon Würstchen). Whereas Stable Diffusion uses a single UNet followed by a VAE, this model uses 2 UNets - resulting in improvements in inference time and training resources.

Rorschach Test For LLaVA LLM

Sat, 30 Dec 2023 00:00:00 +0000

LLaVA works by teaching an LLM (Llama2) to “understand” embeddings from an image model (CLIP). What happens if you replace the embeddings with randomness and ask the LLM questions about what it sees?

Hacking LLaVA

After loading the LLaVA model, we can inject a function that runs after clip embed but before projection to the LLMs token latent space.

# load LLaVA using HF transformers
tokenizer, model, image_processor, context_len = load_pretrained_model(
 "liuhaotian/llava-v1.5-13b",
 model_name="llava-v1.5-13b",
 model_base=None,
 load_8bit=False,
 load_4bit=False)

# inject hook that allows modifying the generated clip embed
tower = model.get_vision_tower() 
tower.register_forward_hook(random_forward_hook)

# if global alter_seed set, override embed with randn of same shape
alter_seed = None
def random_forward_hook(module, input, output):
 if not alter_seed is None:
 print("overriding image embedding with seed: ", alter_seed)
 torch.manual_seed(alter_seed)
 output[0] = torch.randn_like(output[0])
 return output

You can try it out by running llava-lies on replicate. This model still asks for an image, to simplify the modification of LLaVA’s code. The hook random_forward_hook above replaces the output with normalized seeded noise.

Kandinksy 3

Wed, 22 Nov 2023 00:00:00 +0000

Kandinsky 3 is the latest from the team that brought previous versions of the model.

We conducted a lot of experiments on choosing an architecture and did a lot of work with data to make text understanding and generation quality better, and the architecture itself simpler and more concise. We also made our model more “domestic”: now it is much better oriented in the Russian and Soviet cultural field.

Lazy AI Coding Bots

Sat, 03 Dec 2022 00:00:00 +0000

I’ve added a new command to my telegram bot: js. This command will ask GPT3 to generate javascript code and then run it.

For instance if I say ask it to generate a function to reverse a string, it will create function that reverses the letters one at a time.

As GPT3 tended to respond with code that defined functions, I realized I could build up a library of functions! And by sending the list of functions as context for further runs, further functions could be based on existing functions.

Being Misunderstood

Tue, 05 Apr 2022 00:00:00 +0000

I really enjoyed the David Whyte’s Clear Mind, Wild Heart.

About 2/3 in, he said something that reshaped how I go about exploring the world.

Being ok with being misunderstood at the beginning is important. New ideas need to be protected and nurtured.

when you actually tell others what your secret desires are - in your work, in your life.

a good diagnostic feature that you are doing something unique - something that is true to the complexion your character - is that other people should not understand entirely what you are doing

Programmable Cat

Sat, 09 Oct 2021 00:00:00 +0000

Idea

The family was excited to decorate to halloween this year, and several of the decorations came with Try Me buttons. The buttons were external to the decoration - and seem to be simple momentary switches (when they are pressed the circuit is closed for the duration of the button press).

It would be neat to add these to our programmable space using a tiny ESP-8266 microcontroller / MQTT.

Based on success of connecting Miffy to our local programmable space, this seemed approachable for an electronics novice like me.

Lines