Rorschach Test For LLaVA LLM

Sat, 30 Dec 2023 00:00:00 +0000

LLaVA works by teaching an LLM (Llama2) to “understand” embeddings from an image model (CLIP). What happens if you replace the embeddings with randomness and ask the LLM questions about what it sees?

Hacking LLaVA

After loading the LLaVA model, we can inject a function that runs after clip embed but before projection to the LLMs token latent space.

# load LLaVA using HF transformers
tokenizer, model, image_processor, context_len = load_pretrained_model(
 "liuhaotian/llava-v1.5-13b",
 model_name="llava-v1.5-13b",
 model_base=None,
 load_8bit=False,
 load_4bit=False)

# inject hook that allows modifying the generated clip embed
tower = model.get_vision_tower() 
tower.register_forward_hook(random_forward_hook)

# if global alter_seed set, override embed with randn of same shape
alter_seed = None
def random_forward_hook(module, input, output):
 if not alter_seed is None:
 print("overriding image embedding with seed: ", alter_seed)
 torch.manual_seed(alter_seed)
 output[0] = torch.randn_like(output[0])
 return output

You can try it out by running llava-lies on replicate. This model still asks for an image, to simplify the modification of LLaVA’s code. The hook random_forward_hook above replaces the output with normalized seeded noise.

Lazy AI Coding Bots

Sat, 03 Dec 2022 00:00:00 +0000

I’ve added a new command to my telegram bot: js. This command will ask GPT3 to generate javascript code and then run it.

For instance if I say ask it to generate a function to reverse a string, it will create function that reverses the letters one at a time.

As GPT3 tended to respond with code that defined functions, I realized I could build up a library of functions! And by sending the list of functions as context for further runs, further functions could be based on existing functions.

Ai on anotherjesse.com

Rorschach Test For LLaVA LLM

Hacking LLaVA

Lazy AI Coding Bots