If the newness and wow factor of A.I. image generators and Stable Diffusion has worn off for you, deforum animations may be just what you need – While we may not yet be at the point we can scare audiences like the Lumières did with their train video and are probably more in the Magic Lantern stage of development, I had a ton of fun generating the animations for this article.

Deforum Stable Diffusion

Deforum somehow creates animations by creating frames that take their ancestors into account. I do not understand this process at all and haven’t seen a ton of information on it. I will update this if I learn more. It looks to be based on this repo and the papers it links to.

There’s a few ways you can run Deforum SD. As of writing this, Deforum is not available in the Automatic111 repository used in most articles on this site.

• This collab notebook
• Locally using this repo
• Using Visions of Chaos
• This Replicate page

I’m sure there’s other methods too, but this is what I’ve primarily seen people using. We’ll be using Visions of Chaos here which is easy to setup and can run on your own hardware.

Resources and Help

I’ll share some tips around workflows I learned doing some testing, but the best resource I have found to explain the plethora of options in Deforum is Deforum Stable Diffusion Settings. If you’re getting weird results or don’t understand what an option is, check this guide.

There’s also this Google Doc that may be useful.

Check here for other resources and a link the the official Discord server for even more help.

Visions of Chaos

If you’ve heard of VOC, it’s probably as a fractal/generative art making program. They have recently spent a lot of time putting diffusion models in the application and it is awesome and super easy to use. I especially find it useful swapping between post processing tools and models. It’s an awesome piece of software and you should definitely check out what else it can do with its other (non machine learning) modes.

As I write this, I got an update notification they released the Automatic111 webui to VOC. If they keep that up to date reasonably, I will probably switch over to that for ease of use.

You need to enable machine learning in settings and by following this guide to use the SD and other AI features.

Once you do that, you just need to select text to image from the menu:

Then under Script, select Deforum Stable Diffusion v0.4 and you’re ready to go.

Getting Started

The settings are a little confusing to navigate. I won’t walk through everything here but here’s a quick walkthrough to get started.

Prompt - The prompt screen here is not used in animation mode. Use this if you want to test seeds/images when building your animation prompt

Super Resolution - Use this to upscale your images. This will apply to all animation frames if checked.

Iterations - This applies to all frames but you will notice many frames will not take this many steps, assumedly due to how the model works.

Auto save TTI parameter file - I have not used this but if you want to share / recreate your videos, select this to auto save all params.

All other relevant settings are in Deforum Stable Diffusion v0.4 Settings

Animation Mode - If recreating examples below, use 3d

Sampler - I would use the default setting here unless you use euler_a, which may actually produce different results instead of just being slower.

Seed behavior - Since we’re using 3d animation mode, this needs to be set to random or iter not fixed. Fixed will create bloomed geometric animations.

Prompts - The prompts here do not matter for 3d animation mode from what I can see.

Continue to the animation tab.

Use custom animation prompts - Here is finally where we enter our prompt. An example prompt is provided. The numbers are the frames/images that will be generated. Keep in mind the model will try to ‘reach’ the prompt image in the steps it has in-between prompts. If you give it more steps, it will have more time to mutate the image.

Movement Maker you make your prompts, click movement maker and either modify or accept the default settings. The # Key Frames are the # of prompts you have. You may want to change translation_z schedule for zooming in/out.

Now close the settings and hit generate and your frames should start being created. Once everything is done, ffmpeg will launch automatically to create your video.

Tips

• In 3D mode, the translation_z parameter is the zoom schedule and will not be set by movement maker.
• Read over the FOV and translations sections in this guide
• Use lower values in strength schedule for bigger changes in images.
  • In extreme cases you may try using very low strength values for 1-5 frames to dramatically change the image before returning to a normal strength.
  • Example (prompts change at 60 and 120): 0:(0.75),60:(0.5),62:(0.75),120(.55),124(.75)…
• Extend video length using FILM, DAIN, or RIFE in the Movie Interpolation menu after you generate the initial video. This can help you keep processing times down. FILM and DAIN have both produced good results for me. They all have different use cases, so check out each one.
  • If you make your video too long with interpolation, use ffmpeg to double the frame rate (hopefully you started at 30fps)
• If you’re running out of VRAM, see the VRAM offloading setting in the Deforum settings menu. You can also try smaller images using the custom image size dialog.
  • If you have to scale down your images (or even if you don’t), you can use the upscaler in the main text to image dialog and it will apply to all your frames.

Examples

All of these were made in 3d mode. I have not yet experimented with 2d mode.

Anatomy

This example I spent the most time on. Instead of just playing around with random ideas/prompts I tried to create a video I imagined in my mind. You can get the idea from the prompt below.

This example uses an advanced strength schedule referenced above in the tips, where creating the lung, heart and universe were given 40-55% strength values for 1-4 frames to encourage more dramatic transitions.

I also tried setting the prompt schedule to have duplicate prompts to encourage the model to ‘rush’ to a complete version of an object, and then continue fleshing out the idea for several more frames. This worked less well in my experience, but may be part of an ideal workflow – I don’t feel I’ve done enough testing to rule it out.

Frog

Probably the best result I had just randomly playing around. The movement maker settings are default. In this example and the one above, it seems like it would be useful to be able to specify x,y coordinates somehow where the prompt should be rendered, but I imagine that is pretty hard to pull out of a diffusion model in advance. You can likely solve this through camera work most easily.

A Man a Plan a Canal Panama

This was one of my first generations and made with mostly default settings, including all the default movement maker settings. Though the initial image may be odd the results are pretty interesting, though not totally coherent.

Short GIF(like)

I did not have great results with this but I think it could work very well if you spent some time and had the right seed range that didn’t have excessive variations. I actually created this accidentally canceling a longer generation and deciding to generate the video.

Notice the yellow flashing in one of the frames and how that breaks whatever illusion there would be. I think prompt editing instead of completely different prompts may do wonders here. I tested this at the end of this article.

Postprocessing

Built into VOC is a few movie interpolation models. All the above examples use one or a combination of these. Without the interpolation, processing time is long and reasonable fps create jerky videos in most cases I saw. Below are examples of this video processed with the different models.

DAIN

FILM

RIFE

Initial Thoughts

This seems super powerful and as I said before really delivers that wow factor again like the first time you generated a made up idea you had into an image. It doesn’t seem sufficient for really making videos and once you know what these videos look like, it’s pretty obvious when you see a video made with this process.

After glancing at the paper, the main takeaway seems to be the speed at which these generate. Again, I don’t really understand yet how this is all working, but I do notice the iterations seem to vary wildly frame to frame and depend on the amount of ‘changing’ you’re applying via the strength/prompt schedules.

I think all of my initial tests were a little too ambitious too and if you tried to make something like a moving photo all the modern mobile OSs make now, you may have better results. I also wonder what outpainting built into this kind of model could do – especially if it could be contextually aware of other frame expansions.

At this pace, it looks like in 6 months we’ll have on demand feature length films from simple prompts – and somehow maybe we’ll even get to keep some open source options too.

That’s all for now.

If you’ve been keeping up with the latest Stable Diffusion forks, you may have seen prompt editing get merged to the master branch of the Automatic111 repo a few days ago. There’s not a lot of information on how to use this other than a comment in the commit.

I believe the repo this originated from is Cross Attention Control. The README here describes more uses than the AUTOMATIC111 commit comment talks about. I wanted to do some testing on prompt editing anyway and thought I’d see what else out of the Cross Attention Control repo (if anything) worked and how well it performed.

This was one of the most impressive features I’ve tested so far – Here’s a quick peek swapping out prompts. All images are using the same seed:

A whale surfacing	a [whale:starfish:0.1] surfacing	a starfish surfacing

A big limitation of the AI image generators is lack of fine tuning controls. I often get some image that’s close to what I had in my mind, but rather than furiously fighting with prompts for hours, like the poor author of this post, I’ve been waiting to be able to take an image that’s almost what I want and modify within the model it to be closer to my vision.

I think this delivers on that pretty well and has been a lot more impressive than my testing with textual inversion, which has been fairly lackluster but promises similar workflow improvements (by re/teaching the model a concept)

I suspect this is mostly due to poor embeddings – I need to do some more testing on that soon, but I’m starting to think textual embeddings are going to be useful only for representing objects that are too rare or unique to be in the AI model. For most known things/styles, this seems like a more effective approach to fine tuning.

And kudos to the maintainers of the AUTOMATIC111 repo – Even while writing this there have been 10+ new features added including outpainting which I’ve been eagerly waiting for!

Cross Attention Control Functions

The Github repo describes and gives examples of the following uses:

• Target replacement to swap out an object in an image

• Style injection to keep the same base image but modify the tone/artist/etc

• Global editing which is not entirely clear to me and I think is just globally applied concepts that aren’t styles, like weather effects or seasons

• Direct token attention control to boost or deboost tokens/terms in a prompt

• There also looks to be a way to merge images of 2 different prompts together described here.

  I was not able to find a way to do this in the AUTOMATIC111 repo. There is a way to merge images of the same prompt, which we will look at below, but I don’t believe this is part of the Cross Attention Control merge.

From my limited understanding, all of these essentially work generally in the same way. As the image is generated, the prompt is switched out, on the fly, based on your inputs. This could replace a subject or adjective in an image or add additional words to a prompt midway through generation.

Now let’s run some tests.

All test variations will be run from from the same seed and other generation settings. I’ll give some extra information on how I ran the tests in case you want to run your own.

Bomber Object Replacement Test

To start, let’s make a vintage bomber aerial photograph. We can just run some batches of images with random seeds until we get something we like.

After a few generations, we get this:

bomber in flight, wide shot

Now, what if we want the same image but the bomber should be pink? If we just run the same settings and seed and change the prompt, the results are not consistent with our original image:

pink bomber in flight, wide shot

Using Cross Attention Control, we can tell the generation to start with the original prompt we used and switch it at some % through the generation. Here, we’re using 100 steps, so the following prompt will switch to ‘pink bomber in flight’ at 50 steps (50%):

[bomber:pink bomber:0.5] in flight, wide shot

Not exactly what we were looking for, but much closer to our original image and intention. What if we don’t even want a bomber at all. Let’s replace the entire bomber with Paul Rudd:

[bomber:paul rudd:0.3] in flight, wide shot

Awesome. Here are some other style or object replacement tests based on this image. Note, you can add words in addition to replace words, which you’ll see in the prompts below:

[bomber:f16:0.28] in flight, wide shot	[vaporwave:.15] bomber in flight, wide shot	[steampunk:.25] bomber in flight, wide shot	[octane render:.15] bomber in flight, wide shot

Colorful Moon - Token Attention Control Color Test

I did not see Token Attention Control called out in the commit comment explaining how to use this new merge, but it does work. This seems like it may even be a part of some other fork that was merged previously.

In any case, we can take an image and fine tune it by adding + or - to different tokens. Multiple +s and -s can be used for greater effect. This test is pretty simple, but we’ll use this again in the next test with more interesting results.

a red and blue moon

a -red and blue moon	a red and +blue moon

Token control looks pretty hit or miss depending on the token and image. It seems to generally work, but when we ask for more red and less blue below, we get the bluest image of the whole set.

Since we know SD knows what ‘red’ and ‘blue’ are, I assume it does not really understand where the colors are in the image. This isn’t analyzing the image like a Photoshop image filter would by looking at the pixels and I realize don’t really understand what it’s doing or how to explain it failing like this.

a +red and -blue moon

Fruit on the Beach - Putting it all together

Let’s see what we can make with one of my favorite AI Image prompts a giant glittery cantaloupe on the beach at sunset.

Before SD released, most free AI image services couldn’t deliver very well on this prompt, so I kept using it. Let’s use what we learned so far to try to tweak the scene to our liking, swap out the cantaloupe for other fruits, and have some famous artists complete the image.

Here’s our starting image:

giant glittery cantaloupe on the beach at sunset

SD does a good job on the initial prompt but that’s not glittery enough for what I’m thinking.

giant ++++glittery cantaloupe on the beach at sunset

We can do a couple of tests to subjectively boost/decrease the other tokens to tweak even more.

giant +++glittery ++++cantaloupe on the beach at ---sunset	---giant +glittery ++cantaloupe on the +beach at ---sunset

Awesome. Cantaloupes aren’t very popular or recognizable – Let’s try taking the image above, on the right, and using some more common fruits. Note how using different values for when the prompt will change affects the output.

---giant +glittery [++cantaloupe:++apple:.5] on the +beach at ---sunset	---giant +glittery [++cantaloupe:++apple:.2] on the +beach at ---sunset

---giant +glittery [++cantaloupe:++banana:.02] on the +beach at ---sunset	---giant +glittery [++cantaloupe:++banana:.01] on the +beach at ---sunset

For comparison, here are the apple and banana images without prompt switching and using the same seed/settings. They are similar and true to the prompt, but lose a lot from our original image.

---giant +glittery ++apple on the +beach at ---sunset	---giant +glittery ++banana on the +beach at ---sunset

Dali Take the Wheel

We have the pretty good looking glittery apple picture based on our cantaloupe picture above – Now let’s see what Stable Diffusion thinks Salvador Dali would do if we gave him that picture midway through generation.

Note, if we don’t match the prompt switches, he’ll draw us both fruits!

[salvador dali painting of:.2] ---giant +glittery [++cantaloupe:++apple:.2] on the +beach at ---sunset	[salvador dali painting of:.05] ---giant +glittery [++cantaloupe:++apple:.2] on the +beach at ---sunset

Those are slightly interesting, but let’s just give him it from the beginning.

Note, while doing these tests, I randomly tried [term:.00] which should technically be the same as term as it would get added to the prompt at step 0. This does create different images, as you can see below.

Perhaps terms in quotes can’t be added to the first step and this is the same as [term: .01] at 100 steps, but I have not confirmed that.

salvador dali painting of ---giant +glittery [++cantaloupe:++apple:.2] on the +beach at ---sunset	[salvador dali painting of:.00] ---giant +glittery [++cantaloupe:++apple:.2] on the +beach at ---sunset

OK, that looks like a great stopping point. Let’s do some outpainting in DALL-E and use this for the hero image for this post!

Merging Seeds Tests

As I said, while I couldn’t find the tool to merge images of different prompts described here, I did find (I assume from a different merged fork) a way to combine different seeds from the same prompt.

For the Automatic111 repo, this is found by clicking the Extra button next to the seed input in txt2img

To run a test like below, enter a prompt and set a batch size your GPU can handle. Run generations until you find a good base image and modifier image you like. Enter the seeds in the UI and you’re ready to go.

Note, the info output for batch generations appears to be broken and only shows the initial seed, meaning you’ll need to add the image’s position - 1 to the seed to get the correct value. The seed for the image below is 4101654952 = 4101654947 + 5.

Alien Landscapes

Running the prompt an alien landscape with giant blue and orange tropical trees with random seeds, we eventually get these 2 images. Let’s try to merge them to get the look and style of right image with the layout of the left image.

After some failures and settings tweaking, this eventually seemed to work to some degree, though I don’t really see the ‘style’ of trees in the right image in the ‘successful’ generation.

Bangkok

Let’s try a similar test with some AI generated photos. These 2 photos of Bangkok were generated with the same prompt. Can we ‘merge’ them to get a photo that looks like the left image but has the gold structure in the right image?

Not bad. We changed perspective some but I definitely think both images subjects and ‘feel’ are represented in both images.

I tried playing with the scale sliders for the variation to try to shrink the gold structure, hoping it would be somewhere else in the background. This kind of worked. It looks like it shrunk the structure, but not as I intended. It wants to put it on top of a skyscraper regardless of settings.

Final Thoughts

For a few quick tests, I think these results are very impressive. I’m sure with a bit more time, effort, and refinement you could get extremely close to the image or modifications you’re looking for. Share your tips below in the comments.

Next time we will hopefully look at textual embeddings and outpainting in Stable Diffusion, which released yesterday.

Wow – The pace at which the AI Image generation space has changed is something I have seldom seen in my life as a nerd. It reminds me of early Bitcoin tools or the Apple app store when it seemed like every day there was some new amazing app/tool/offering.

I think SD is even more engaging for the community than these since it’s both open source and produces something tangible and fun (looking at blockchain ledgers takes some imagination to see why it’s ‘cool’)

I personally had so much fun hacking on the original CompVis repo and building my own workflows. Even after a good amount of work though, the way I had been using Stable Diffusion wasn’t really sufficient for anything other than playing around or running tests on prompt syntax.

If I wanted to use SD as an artist and bring images to life based on my creative vision, I was missing a UI, simple integration between txt2img/img2img/face restoration/upscaling/etc, and so much more I didn’t even realize. Thankfully, a group of people much smarter than me have already got together and built all this out.

You can find the Automatic1111/stable-diffusion-webui repository here

This tool really lets you peek into the power of what these kinds of tools will do for artists in the very near future. The repository is updated frequently with new features or tools too – below we’ll look at setting up textual inversion.

Install

Installation is so simple and covered so well in the repository, I won’t add anything here. After installing the right version of Python, simply clicking the .bat file will install everything you need except the SD ckpt model. The model download links are also located in the repo.

Extras

I’d recommend installing GPFGAN and ESRGAN as described here. This will let you fix faces and easily upscale images. CodeFormer is also included but will be downloaded when you attempt to use it.

Making it Public

If you’re reading this, the likelihood you’ve been playing with AI image generators and SD for the past weeks/months is high. If you’ve been running SD on your machine, you almost certainly have wanted to show your friends too – Kudos to you if you exposed your machine publicly and walked people through entering terminal commands (I guess :P) – Now, thankfully, we have an easier way.

If you want to launch stable-diffusion-webui via a public URL in addition to your localhost url, you can make a simple edit to webui.bat:

Find the line that starts with launch: and change it to the following:

:launch
echo Launching webui.py...
%PYTHON% webui.py --share %COMMANDLINE_ARGS%
pause
exit /b

Now, when you launch webui-user.bat, a public URL will appear in the terminal you can share with others

Textual Inversion

Textual Inversion is a very new concept in AI Image Generation/Stable Diffusion and allows you to create concepts and pass them to the model via embeddings. This means you can create databases of images that represent different styles or concepts and use them in your image generation prompts.

The Automatic1111 repo lets us do this today! Let’s set it up.

You can read more about textual inversion here

Setup

Per the repository, we need to create an embeddings folder in the repositories root folder.

Once we do this, we need to get some embeddings. This will almost certainly change as I write these words, but currently, there is a repository of user submitted embeddings available through HuggingFace/sd-concepts-library.

Find some embeddings you like here. Download the learned_embeddings.bin file in the repository:

From my brief research, a lot of these look to be comprised of only a few images. If you look for larger bin files, you can semi-easily see how large an embedding’s dataset is. I found larger datasets produced better results in the few tests I ran.

Rename the learned_embeds.bin file to name.pt where name is the term you want to use in your prompts to refer to the embedding.

You must not have folders or files other than the .pt embedding files in the embeddings folder or you will see errors in terminal.

If everything worked correctly, when you run a txt2img prompt with the embedded term, you will see “used custom term” in your detail output.

Final Thoughts

Very excited to see what new features come out in the near future. I have been playing with the tiling settings I didn’t have access to in the CompVis repository with some really cool results, which I’ll share later this week. Here’s a sneak peek of some of the final results:

I’m also very keen to generate my own embeddings and do some more testing to better understand how that works and can be used.

That’s all for now – Enjoy!

Stable Diffusion Release!

We knew the world would not be the same – A few people laughed, a few people cried, most people were silent.

And just like that, the whole world changes. Stable Diffusion and its openness is a massive improvement to the tools we’ve been using in this space and our ability to modify/understand them.

It was bound to happen, but I am very impressed with both how quickly it rolled out and how powerful the tool is.

I still have not explored most of the models or played around too much with the new tools added since latent diffusion, but one thing that was hindering me from doing testing at a good pace in the Latent-Diffusion repo was some challenges with the scripting and how the images are output.

No one likes long terminal commands and digging through integer named files – I decided to do something.

As often happens with these kinds of things, what started as a simple exercise to build a bash alias ended up in a 2 day journey battling shell syntax and figuring out how to automate image processing tasks.

Making Improvements

I’ve been messing with improving the scripts and tools in latent-diffusion for a couple of weeks. Most of the things I was trying to solve for or had solved still existed in the Stable Diffusion release.

I was able to port some of the changes I made to latent-diffusion to stable diffusion and decided to put some more effort into making simple ZSH commands to run bulk tests for txt2img and img2img and will release them below. Improvements include:

• Sample file names include prompts
• Easier to run with defaults or parameter changes (btxt2img 4 42 100)
• Unique folder output based on ISO dates
• Better grid creation with imagemagick
  • img2img grid includes original photo and prompt labels for each sample
  • txt2img grid includes prompt labels and 1 row per prompt
• Grid settings (fonts, colors, borders)

Stable Diffusion Helper Scripts

Stable Diffusion Helper Scripts requires ZSH. I have not tested this in bash, but I suspect a bunch of things will break. If you want to port this to bash or anything else – go for it and please let me know – but, you should probably just download ZSH if you intend to use this.

All the files you need are located in the Github repository.

Instructions and Examples

Img2img Grids with Different Strengths

Txt2img Grids

How to Use

• Clone the repo or download the files
• Copy txt2img2.py and img2img2.py to your stable-diffusion/scripts folder
• Run cat ./zshrc_scripts >> ~/.zshrc
• Open ~/.zshrc in a text editor and fill out the User Entered Variables section
  • The paths for Stable Diffusion and the font to use for grid labels need to be changed for your system
  • You will need to create the prompt txt files if they do not exist
• Restart your shell or run source ~/.zshrc

Now you can create a list of prompts in your img or txt prompt file (1 prompt per line like with the standard script).

When you’re ready to generate, run btxt2img or bimg2img /path/to/img.

The samples (and original file for bimg2img) will be saved to its own timestamped directory in the output folder. The new grid will be saved to the same directory as fullgrid.png

For help changing parameters, check the ~/.zshrc file or run either command with the --help argument.

Up Next

In Latent-Diffusion, I created a script to compare RAD with LDM models that I need to remake in Stable-Diffusion. I will probably wait to do this until I figure out how to make my own image databases for RAD. I did find a repository that has an example of doing this, so hopefully I can use that to figure something out.

I also need to rework the grid tools to account for generations of more than 5 or 6 images to split the strips into multiple rows.

Enjoy – Please leave any questions or comments below!

In this post, we’ll look at getting setup with running your own A.I. image generator. You need a Linux system with a CUDA enabled card to get this working through to the end. I did get the basic txt-2img script working in Windows. Unfortunately, for the July released retrieval-augmented diffusion models, you need scann in order to index the openimages dataset, which is only available on Linux. WSL worked fine for me.

Here’s roughly what I gather you’ll need:

• Linux (WSL fine)
• 11.3 CUDA graphics
• >= 32GB RAM
• 50GB disk space (~30GB without openimages)
• 3xxx series card (I am just guessing here. This is all being done with a 3090 FE)

Latent-Diffusion

I randomly stumbled upon the latent-diffusion repo while I was browsing AI subreddits. A few posts mentioned it and I didn’t know what it was so I went googling.

I am not a computer scientist, mathmatician, or any of the things that would be required to really understand what’s going on here. The paper about the new RAD models are understandable and interesting, but I haven’t done much research yet beyond that.

Text-to-image vs. RAD

There are 2 image generation techniques possible with Latent Diffusion. We will install and take a look at both.

I believe the txt2-img model that we’ll setup first is what we are used to with other image generation tools online – it makes a super low res image clip thinks is a good prompt match and denoises and upscales it.

The RAD model uses a configurable database of images as a reference AND does the diffusion like we are used to. This seems to mean it’s less good as a general purpose generator, but could be used with a specific training set you could feasibly create. I will try this in the future.

RAD also generates 768X768 images which is pretty impressive.

Install Instructions

Text-to-Image

• Make sure you have CUDA 11.3 installed and conda works in your shell. You can get Conda here

• (Re)install Pytorch now, lest ye end up in dependency hell later. I did this via pip but conda may work too.

• Clone the repo to a drive with a lot of free space

• cd into the repo and run:

  conda env create -f environment.yaml
  conda activate ldm
  

  This will setup the initial environment. If you get errors with Pytorch later and need to reinstall it, come back to environment.yaml and reinstall using the correct package versions.

• Download the model and you’re all ready to go so long as your torch dependencies are all setup correct.

mkdir -p models/ldm/text2img-large/
wget -O models/ldm/text2img-large/model.ckpt https://ommer-lab.com/files/latent-diffusion/nitro/txt2img-f8-large/model.ckpt

You should be able to sample images now using:

python scripts/txt2img.py --prompt "my cool image"

Make sure to see the options you’re able to set in the README.

You can use this bash script to make running everything from a shell easier. You can include this in your ~/.bashrc file and reopen your shell. Make sure to change the 3 paths to suit your needs. We’ll look at PLMS later.

function txt2img() {
  if [ $# -eq 0 ]; then
    echo "Usage: diffuse '[prompt]' [--plms]"
    exit 1
  fi
  cd /mnt/c/diffusion/latent-diffusion;
  conda activate ldm;
  if [ $2 = '--plms' ]; then
    python scripts/txt2img.py --prompt $1 --outdir '/mnt/g/My Drive/AI Image Tests/text2img' --plms --n_iter 4 --ddim_eta 0.0
  else
    python scripts/txt2img.py --prompt $1 --outdir '/mnt/g/My Drive/AI Image Tests/text2img' --n_iter 4 --ddim_eta 0.0
  fi
}

The images are 512X512 by default. You can use This Jupiter Notebook to upscale the images with pretty good results.

Troubleshooting

• If you get CUDA memory errors running commands, try decreasing the batch size. Use htop to troubleshoot/view memory usage.
• If using WSL, make sure all your RAM is available to linux.
• If you get torch related errors, you probably need to reinstall Pytorch. This will likely cause other issues with packages. Check environment.yaml and manually install the correct package versions with pip and/or conda (I did both and am not sure which made everything work)

RDM

You can read more about RDM here. As said before, it essentially combines a database of images with Clip descriptions with the existing diffusion process, from what I understand. This part takes a lot longer to get running.

Note, this model, especially when using the openimages training is best at recreating real things and doesn’t seem very good (yet) and creating the weird images we’re used to from the diffusion models.

• Get the Text-to-Image model working first

• Install new packages and download training model.

  pip install transformers==4.19.2 scann kornia==0.6.4 torchmetrics==0.6.0
  pip install git+https://github.com/arogozhnikov/einops.git
  
  mkdir -p models/rdm/rdm768x768/
  wget -O models/rdm/rdm768x768/model.ckpt https://ommer-lab.com/files/rdm/model.ckpt
  

• Test everything is working so far with

  python scripts/knn2img.py  --prompt "I'm a computer"
  

• If everything went well, you should see a success text in your shell. Now we need to download all the image indexes/models. The openimages zip is 11GB. The ArtBench data is pretty small.

  mkdir -p data/rdm/retrieval_databases
  wget -O data/rdm/retrieval_databases/artbench.zip https://ommer-lab.com/files/rdm/artbench_databases.zip
  wget -O data/rdm/retrieval_databases/openimages.zip https://ommer-lab.com/files/rdm/openimages_database.zip
  unzip data/rdm/retrieval_databases/artbench.zip -d data/rdm/retrieval_databases/
  unzip data/rdm/retrieval_databases/openimages.zip -d data/rdm/retrieval_databases/
  
  mkdir -p data/rdm/searchers
  wget -O data/rdm/searchers/artbench.zip https://ommer-lab.com/files/rdm/artbench_searchers.zip
  unzip data/rdm/searchers/artbench.zip -d data/rdm/searchers
  

We’re ready to use the Artbench models now (which work pretty well in my limited testing), but what we really want is to use the massive openimages model as our reference. We downloaded the data, but we need to create the index.

If you want to test the Artbench database, run

python scripts/knn2img.py --prompt "A blue pig" --use_neighbors --knn 20 

Openimages Index

Unless you have a super computer, the terrible memory management in python multiprocessing (from what I can tell) will stop us from using the 4 files we unzipped with the script from the repo to index them. Everything else I tried, the processes would run out of memory.

Luckily, we just need to concatenate the files. You can delete the old part files after if you would like. We also need to move the part files out of the openimages folder.

cd data/rdm/retrieval_databases/openimages/
cat 2000000x768_part1.npz 2000000x768_part2.npz 2000000x768_part3.npz 2000000x768_part4.npz > 2000000x768.npz
mv 2000000x768_* ~/temp
cd ../../../../

Now when we run the training script, we shouldn’t run out of memory. I think this is some issue in their multi file function in the training script.

python scripts/train_searcher.py

• We need to change the batch size in scripts/knn2img.py so we don’t run out of GPU memory. You may need to lower to 1, but I was able to set to 2 on a 3090.

• Open scripts/knn2img.py in an editor

• Go to line 243 or find “n_samples”

• Change the default value to 1 or 2 and save

• You can also do this using the command line parameter, but since it will never work for me, I wanted to change the default value. Note if you update the repo in the future this will likely get overwritten.

Now we can generate RDM images using the openimages database:

python scripts/knn2img.py --prompt "A blue pig" --database openimages --use_neighbors --knn 20 

Here is a ~/.bashrc script to run it conveniently. Again, make sure to change the 3 paths to suit your needs:

function diffuse() {
    if [ $# -eq 0 ]; then
       echo "Usage: diffuse '[prompt]' [--plms]"
       exit 1
    fi
    cd /mnt/c/diffusion/latent-diffusion;
    conda activate ldm;
    if [ $2 = '--plms' ]; then
        python scripts/knn2img.py  --database openimages --prompt $1 --use_neighbors --outdir '/mnt/g/My Drive/AI Image Tests/RDM' --knn 20 --plms
    else
        python scripts/knn2img.py  --database openimages --prompt $1 --use_neighbors --outdir '/mnt/g/My Drive/AI Image Tests/RDM' --knn 20
    fi
}

Speeding Things Up

You can also run prompts in batches by creating a file (in this case prompts.txt) and pointing the script to it with the –from-file parameter like this

python scripts/knn2img.py  --database openimages --from-file ./prompts.txt --use_neighbors --outdir '/mnt/g/My Drive/AI Image Tests/RDM' --knn 20

sample prompts.txt

prompt 1
prompt 2
prompt 3

This should save a lot of time if you’re running a lot of queries since the model only gets loaded once. The text-to-image model does not have this option.

Initial Testing, & PLMS

Below are some inital results from both the models with PLMS on and off. PLMS should make things run faster. For now, we won’t measure the speed increase but will look subjectively at image quality/results.

Since the RDM model will only make 2 photos at a time for my system, I included both photos instead of a grid for that model.

I will almost certainly do more specific testing in the future and these models will be included in future comparison posts.

A wood house on a hill, landscape photography

Text-to-image

without plms	with plms

RDM

without plms	with plms

A purple stop sign

Text-to-image

without plms	with plms

RDM

without plms	with plms

fox den, digital art

Text-to-image

without plms	with plms

RDM

without plms	with plms

winning the big game, award winning photography

Text-to-image

without plms	with plms

RDM

without plms	with plms

a sports car driving in sand dunes

Text-to-image

without plms	with plms

RDM

without plms	with plms

An Image Generator to Call Your Own

And there you have it – no more credits or monthly limits, just pure AI image generation all your own. Once I figure out some tuning settings and create my own image training database, we’ll likely talk about this again.

Hopefully with some tuning and new releases/models this will get closer to the WOW factor DALL-E has.

Please share any tips, questions and creations in the comments!