Increasing Imagination 💡
Life, Imagination, Creation ft Aqua
Imagination. Life is your creation - Aqua (Barbie Girl, 1997)
The promise of text-to-image AI models like Midjourney is to make the world more beautiful.
I want to see a more beautiful world, but I’m even more excited about a world with more imagination. What would an AI model that increases the vividness of our imagination look like?
1. Life
I’m particularly interested in the question of how Imagination leads to Creativity since, depending on how you look at things, I’m either quite creative or not creative at all:
I used to be a musician - performing, designing sounds for synthesisers and writing songs.
But, I’m terrible at drawing. One time at school, a teacher held up everyone's drawings and skipped over mine. She later apologised, sheepishly admitting that she wasn’t sure whether to announce that my drawing was a tractor or a tree. It was supposed to be a cat.
Aphantasia is a trait where some people can't visualise objects in their heads. If you ask someone with aphantasia to picture something in their heads, they literally cannot do it. They just don't think in photorealistic pictures at all.
If you ask an aphantasiac to describe the Mona Lisa they might tell you some facts they remember (there’s a brown-haired woman) or maybe some remarks they remember people talking about (her smile is weird). But they’re not doing that because they can visually see the image in their mind, they’re recalling events, facts or conversations relating to the Mona Lisa. They’re thinking in narratives or concepts but never in pictures.
To many non-aphantasiacs, this feels almost unbelievable - how can they even function without having a ‘mind’s eye’?
I have aphantasia. Although it hasn’t really affected my life, it’s hard to think of a bigger handicap for making art - it's super hard to draw from memory because I don’t imagine some visual concept to draw out. (For reference I had to look up the colour of the Mona Lisa’s hair to write this and I had no recollection of what the background is at all.)
It took me a while to realise that when people talk about ‘picturing’ something or conjuring up an image, they don’t mean it metaphorically. To me, seeing images in your head sounds like a superpower.
I recently did an introspective experiment to understand if I could see some visual images in my mind, even if only very slightly. Here’s what I found…
2. Imagination
To create is human. But to imagine? Oh honey, that’s divine - Anon
First, let’s say what we mean by creativity.
Scottish philosopher David Hume writes that the “creative power of the mind amounts to no more than the faculty of compounding, transposing, augmenting, or diminishing the materials afforded us by the senses and experience”1. Note that here he’s describing creativity in the mind and not out in the world. It’s like judging a painter's creativity by their vision for their artwork rather than their actual paintings.
In other words, creativity is combining things in a new way - especially if it’s useful, beautiful, or admirable. Creative acts can be new for you (like a shower thought), new for the whole world (like proving a new mathematical theorem), or something in between.
Creativity always recombines existing elements. There is nothing new under the sun2.
So let’s break down the Creative Act into parts:
Life - Our sensory experiences act as inspiration for art3. For example, you might go to a museum to get inspiration for your painting
Imagination
Recombination - Deciding how to combine, augment and filter your life experience into an idea. For example Van Gogh was inspired by French impressionist painter Camille Pissarro to use brighter colours and shorter brushstrokes. And he was also inspired by Japanese ukiyo-e woodblock prints to use 2D perspectives and bold outlines. We corrupt and process images (perhaps in a way that’s totally unrecognisable from the original) and filter the latent image through the prism of our taste4.
Representation - Representing the idea in high fidelity in your mind’s eye. Once you have an idea you want to understand the concept vividly.
Creation - Using your hands to get the idea in your head onto the canvas5. Actually painting the picture. Creation is the physical, mechanical act of reproducing what’s in your mind.
Note: This doesn’t have to be a linear process. You can realise as you’re drawing you need more inspiration and do more research. This would be looping back to step 1 (Life).
Or you can reimagine your idea based on what starts coming out on the page - some of the ideas of a painting come to you whilst you’re making it. This is looping back to step 2 (Imagination)
Like Hume above, note the distinction between getting an idea into your mind and getting the idea from your mind into the world.
Indeed, we might think that putting an idea into the world isn’t necessarily creative. Being able to implement a clear vision doesn’t seem like creativity in the normal sense.
For example, if someone draws a perfect replica of an image that’s in front of them, I’d call them talented but I’m not sure I’d call them imaginative or creative. The imaginative person seems to be the one who draws something that we’ve never seen before.
In this sense, the truly imaginative steps of creation seem to be Recombination and Representation. Mechanical creativity is more like fine motor skills - impressive but not inherently creative.
Most art classes teach mechanical creativity - they teach about brushstrokes, perspective, and styles. They’re teaching getting your idea onto the canvas. It’s harder to teach imagination - getting the idea into your head in the first place.
Text-to-image AI helps with mechanical creativity. If you have an idea, Midjourney can help you to instantiate it. Is it possible for an AI to help you with the Imagination part?
3. Creation
There’s one deep problem with judging painters by their vision rather than by their output, à la Hume. There’s no way to see their vision, except through how they translate this vision onto the page; we cannot, unfortunately, see inside their heads.
This isn’t just a problem for people judging painters - it’s a problem for the painter themselves too. We cannot see inside our own heads either, not really.
If you’re trying to improve as a painter, how can you concretely see your imagination improving over time if you don’t have sketches over that time period that you can tell are getting better?
Drawing bicycles
There’s a classic study where participants are asked to draw a bicycle from memory. They’re told to focus on the elements which will allow the bicycle to function properly e.g. the frame, pedals and chain.
In theory, this is pretty easy - we’ve all seen thousands of bikes in our lives and whenever we see one, we have no problem at all recognising it as a bicycle.
In practice, however, well…
Most of the outputs wouldn’t be rideable. Most had a chain placed incorrectly or dimensions that were way out of whack. What’s going on here?
The reason is expressed by a learning mantra: Evaluation is easier than Generation. Let’s break that down. Being an armchair critic (evaluating a performance) is much easier than being a football player (generation, having to actually play for yourself). You need to have a much clearer idea of a bicycle in your mind to draw (generate) a bicycle than to recognise one (evaluate whether it is a bicycle or not).
Having some fuzzy, low-fidelity, cropped image of a bike in your mind isn’t enough to successfully draw one. But it is enough to say whether a random image is a bike or not.
So if we wanted to actually build a bike that worked, we need a clearer, more vivid model of a bike in our minds. It’s not enough to see bikes, we must practise drawing them. We have to generate.
Imagination Practice Game - From Life To Imagination To Creation
This suggests an Imagination Practice Game6:
🧑🎨 Draw a bike from memory.
↔️ Compare it against a real bike and note where the drawing isn’t quite right.
♻️ Try another drawing incorporating the improvements and see if it’s any better.
🚲 After a few tries of this, we should get something that looks very much like a realistic image of a bicycle.
Here we isolated the mental task of drawing from the physical task of drawing7. The drawing didn’t get better because our execution got better. It got better because we had a clearer mental idea of what we wanted to achieve - we got a clearer idea of a bike in our minds.
I tried this and even though I’m not necessarily seeing the bicycle in my mind visually per se, I did get a little better at being able to draw from memory. This indicates that I’ve gained some sort of mental representation of a bicycle that’s slightly stronger than it was before!
I just trained my imagination.
Human Learning From AI Feedback
This method for improving imagination seems to work, but there’s one step that takes a long time - actually drawing out the image to compare it against the picture. We’d really like to have faster feedback loops.
I recently saw a relatively unhyped AI from Meta. Instead of text-to-image, this model was brainwaves-to-images. You think of an image and the AI draws out an image representation of your brain state. (It’s all non-invasive, you don’t have to put anything inside your skull or anything8).
Right now, this is fairly primitive and doesn’t totally work. But imagine if it did. I think the most exciting application for this would be speeding up our Imagination Practice Game.
Given an image, you should hide it and imagine it in your mind.
Let the AI instantiate the image that you imagined on a screen.
Now look back at the original image and compare the differences.
Repeat the process. Compared to our original Imagination Practice Game, here each iteration goes from taking minutes to seconds.
When we teach an AI to learn based on humans’ ratings, we call it Reinforcement Learning from Human Feedback (RLHF).
Here we have the opposite. The AI system is giving feedback to us - telling us how similar the image in our mind is from the true image - allowing us, as humans, to learn more effectively. Perhaps we might call this Human Learning from AI Feedback (HLAIF).
This Human Learning from AI Feedback approach has the potential to be an important tool for training creative artists. We can directly train our imagination rather than only improving via our technical abilities.
An early example of a game to take inspiration from here is MindLight, a therapeutic children’s game for reducing anxiety. While the child plays, they wear a headset detecting brain waves and can recognise if the child is getting stressed. The game then allows the child to recognise this emotion and the in-game dynamics offer positive reinforcement to conquer their fears (and the in-game monsters 👻). As it turns out, playing this game instead of a CBT therapy session had comparable effects, reducing anxiety even 6 months after the game sessions!
I’m envisioning a MindLight-style game that, instead of reducing anxiety, increases imagination through rapid feedback and iteration with a brainwaves-to-image AI.
Even with Human Learning From AI Feedback for Imagination, I’m not sure I’ll ever have a super precise visual imagination. Right now, there’s a huge difference in imagination between me and the average person.
After HLAIF, I wonder if top artists could increase their level of imagination by a similar difference. Looking back at their old imagination, might be like the average person looking back at my visual imagination: “How could they even function?”, they’ll ask.
Perhaps our era might be the most imaginative of human history so far.
Imagine that.
Thanks to Andrew, Derik, Sarah, Diego, Joe, Hans and Lydia for comments on the talk which inspired this post. And thanks to Anna, Gad, Matt, Emma, Vijay, Paul and Garaminder for reading early drafts of this essay.
Famously Steve Jobs simplifies this sentiment to “Creativity is just connecting things”.
Indeed improvements in Artificial Intelligence have shown this to be true in a concrete sense - you really can get amazing art "just" with a model that understands and recombines the data it was trained on.
Incidentally, this points towards an interesting implicit definition of beauty. An object is beautiful to the extent that it inspires imagination. The beautiful object is the one that wills you to reproduce it in your mind.
The canvas need not be visual, the same holds for a musician whose canvas is a recording.
Machine Learning people know this trick for training AI systems - we changed our approach from classification to autoencoding reconstruction.
Note that if the picture in your head was perfectly clear, then the task of drawing wouldn't be some mysterious and difficult task of assembling parts together and thinking things through carefully - it would be just like drawing a still life that's sitting right in front of you. So this task being hard (and even harder for aphantasiacs) shows that even though we think we have a clear visual idea of a bicycle in our heads, we really don’t!
cough, Neuralink cough