|

How To Generate Consistent Faces In Stable Diffusion

If you’re an AI enthusiast, I’m sure you must have come across videos on YouTube about how someone made an AI influencer and is now raking in thousands of dollars.

Such videos are all over my feed and while I don’t know how truthful they are, it still made me curious about how they’re doing it. 

Generating consistent faces in Stable Diffusion is not that easy and if you want to create an AI character, you’ll have to achieve consistent faces in all the images you generate. 

Whether you want to create AI influencers or just want to turn yourself into a superhero, I’ve got the ultimate guide here for you. 

In this guide, I’ll show you the different methods to generate consistent faces in Stable Diffusion. 

I’ll walk you through each method and give tips and tricks to ensure you generate the same face without much hassle. 

That being said, let’s get started. 

How To Generate Consistent Faces In Stable Diffusion

There are many methods to generate consistent faces in Stable Diffusion and instead of covering only the best one, I think it’s better to share them all here. 

This will help you understand all the methods and decide for yourself which one is suited better for your needs. 

But I’ll begin with my favorite and recommended method that has helped me create consistent faces in Stable Diffusion. 

Note: Before we begin, I want to point out that the techniques shared in this guide are powerful and allow face swapping in a matter of clicks. That being said, we do not recommend or encourage the use of real-life people/celebrity faces to create any inappropriate or demeaning images.

Method 1: Using ControlNet IP Adapter Face Models (Recommended)

The best method to get consistent faces across all your images is to use the ControlNet IP Adapter. ControlNet is a neural network model used in Stable Diffusion to influence image generation. 

Think of it like LoRA models but more advanced and with a lot of refinements. There are a lot of ControlNet models for different use cases and IP Adapter models for copying faces from an image. 

With IP Adapter face models, you can use a face from your input image and copy that face to your generated image. 

Let’s see how it’s done. 

Setup ControlNet & IP Adapter Models

Step 1. Make sure ControlNet is installed in your Automatic1111 WebUI. If not, go to the Extensions tab in Automatic1111 and select Install from URL

Automatic1111 - Install Extension from URL

Then, enter this URL to install ControlNet – https://github.com/Mikubill/sd-webui-controlnet

Once, ControlNet is installed, restart your WebUI. 

Step 2. Download the IP Adapter face models from the HuggingFace website. Download the ip-adapter-plus-face_sd15.bin model from this page. 

Download IP Adapter Models

Step 3. Place the downloaded model in the stable-diffusion > stable-diffusion-webui > models > ControlNet directory. 

Consistent Faces - Place IP Adapter Model In Directory

Using IP Adapter Model In ControlNet

Step 1. Open Automatic1111 and go to the ControlNet in the txt2img tab and enable ControlNet. 

Consistent Faces - Enable ControlNet

Step 2. Import the image from which you want to copy the face. 

ControlNet IP Adapter - Import Image

Step 3. Select the IP Adapter in the Control Type settings. Select ip-adapter_clip_sd15 in the preprocessor dropdown and our downloaded model ip-adapter-plus-face_sd15 in the Model dropdown. 

ControlNet IP Adapter - Select Models

Step 4. Set the control weight to 0.5 and ignore the other settings. 

Consistent Faces In Stable Diffusion - IP Adapter Settings

Leave the other settings as they are for now and scroll up to write the prompt for your image. 

Generate Your Image

Step 1. In your image generation settings, enter a positive and negative prompt and choose your checkpoint model. I’m using the epicphotogasm checkpoint model. 

Automatic1111 - Select Checkpoint Model

Here are the prompts and image generation settings: 

Positive Prompt: 

portrait photo of 1man, superman costume, closeup, cinematic

Negative Prompt: 

blurry, deformed, text, ugly, cartoon, illustration

Generation Settings: 

  • Image Size: 512x768px 
  • Sampler: DPM++ 2M SDE Heun Karras
  • Steps: 30 
  • CFG Scale: 5

Step 2. Click on the Generate button to begin image generation. 

Here’s the generated output: 

You can experiment with the control weight and starting/ending steps to get a more accurate output for your face. 

I’ve experimented with control weights and found a weight of 0.5 to be perfect and most accurate to the input image. But this can depend on your input image and your prompt. 

ControlNet IP Adapter - Control Weights

Moreover, you’ll also find the images to not be of very high quality. That’s how it is when you’re swapping faces in Stable Diffusion. 

To fix that, you can simply upscale the image using a hires fix or upscale them in the Extras tab. 

With IP Adapter models, you can easily copy faces and generate new images using them. This is perfect for generating consistent faces for AI characters, influencers, or anything in between. 

There are other IP Adapter models as well that you can try for both SD1.5 and SDXL

I’ve also used the IP Adapter SDXL model for copying faces in ComfyUI using this workflow which I found better than the SD1.5 IP Adapter models.

You can download this workflow and use it in ComfyUI or use the IP Adapter in Automatic1111 according to your preference. 

ControlNet IP Adapter - ComfyUI Workflow

I prefer using ComfyUI since it’s faster than Automatic1111 and gives a bit more refined control over each step of image generation. 

Related: How To Change Clothes In Stable Diffusion

Method 2: Using ReActor Extension 

The second method to generate consistent faces in Stable Diffusion is to use the ReActor extension. 

This is a face swapping extension that allows you to swap your face to images. It works similarly to ControlNet IP Adapter models. 

Let’s see how it works. 

Installing ReActor Extension 

Step 1. In Automatic1111, go to Extensions > Install from URL and enter the following URL: https://github.com/Gourieff/sd-webui-reactor

Automatic1111 - Install Extension from URL

Step 2. Once installed, restart your Stable Diffusion WebUI and you’ll find a ReActor extension toggle in the txt2img tab. 

Automatic1111 - ReActor Extension

Note: Many users have trouble installing this extension due to errors and if you face the same, check out their installation guide which goes through the steps in more detail and has also laid out some steps to troubleshoot any errors you face during installation. 

Swapping Face Using ReActor 

Step 1. Open Automatic1111 SD Web UI and go to the ReActor in the txt2img tab and enable it. 

Automatic1111 - Enable ReActor

Step 2. Upload the image you want to use for copying the face. 

Automatic1111 - ReActor Upload Face

Don’t change any settings in the ReActor tab and scroll up to write your prompt. 

Generate Your Image

Step 1. Enter a positive and negative prompt. Here are the prompts I’ve used along with the image generation settings: 

Positive Prompt: 

portrait photo of 1girl, closeup, upper body, night city, outdoors, bokeh, blonde hair

Negative Prompt: 

blurry, deformed, text, ugly, cartoon, illustration

Generation Settings: 

  • Image Size: 512x768px 
  • Sampler: DPM++ 2M SDE Heun Karras
  • Steps: 30 
  • CFG Scale: 6

I’m using the same checkpoint model as I did for the IP Adapter. 

Step 2. Click on the Generate button to begin image generation. 

Here are the output images: 

The face swapping is very accurate but the images are way too blurry to be useful. 

So, let’s go back to the ReActor extension settings and select the CodeFormer option under the Restore Face settings. 

ReActor - Enable CodeFormer

Generate the images again and you’ll notice the blurriness has now completely gone. 

You can also try using GFPGAN for restoring faces as it also has pretty good results. 

Overall, ReActor is a very good extension for generating consistent faces in Stable Diffusion. The only reason I recommend the IP Adapter over this is because ReActor uses a 128px model for swapping which causes blurriness. 

Of course, you can fix the blurriness by selecting the Restore Face option. But I still prefer the IP Adapter method. 

However, if you’re struggling to run ControlNet on your system, then ReActor is the best option out there for face swapping and getting consistent faces. 

Method 3: Using LoRA Models

The next technique for generating consistent faces in Stable Diffusion is by using LoRA models. 

There are plenty of LoRA models out there of famous people and celebrities. In fact, I even shared a list of the best celebrity face LoRA models a while back. 

So, if you want to get consistent faces to well-known personalities, you can find their face models on Civitai. 

Here’s an example of a LoRA face model of Dwayne Johnson:

However, if you want to use your own face or someone you know, you’ll have to train a LoRA model of their face. 

Creating LoRA models is not very easy but this guide is a good starting point. You’ll need a lot of images of the person whose LoRA model you want to create. 

This is not an easy approach and unless you want to generate consistent faces for professional purposes, I wouldn’t recommend going this route as it’s very complicated. 

Method 4: Combining Celebrity Names

You might be aware that writing a celebrity’s name in your prompt can help you generate a face that almost looks like it. 

But do you know that you can combine two celebrity names and get a new face? 

Not only this, you can then combine those two names and generate that one face consistently. 

I was skeptical when I first read about this various times on Reddit. So, I decided to try it out and the results are quite promising. 

Here’s how you can combine celebrity names to generate consistent faces. 

Let’s write a prompt that combines two celebrities to get a new face. 

Here’s the positive prompt: 

photo of a (Jenna Ortega | Eva Green), smiling, sitting on a sofa, closeup, upper body, smile

I’ll be using the epicphotogasm checkpoint model for this generation. Here are the output images: 

As you can see the output face looks like a mashup between Jenna Ortega and Eva Green. 

This is impressive but what if you want to generate this same face in a different setting or environment? 

Well, you can do so by using the same celebrity name combination like this: 

Here’s the prompt I used to generate the images above: 

photo of a (Jenna Ortega | Eva Green), standing, wearing a long wedding dress, garden, upper body, closeup

As you can see, the face looks similar to the one we generated before. 

I like this technique for generating consistent faces but it’s only good as long as you’re generating closeup shots. 

As soon as your prompts get complicated, the faces start to look different. Still, it’s a good technique if you want to generate consistent faces without using ControlNet or any other extension. 

Method 5: Using Random Names 

The last method to generate consistent faces in Stable Diffusion is quite unconventional but works great. 

In this technique, you use a random name in your prompt and it’ll generate a similar face every time you use that name. 

Here’s an example image where I used the name “Ailee Cortez James” in my prompt: 

Prompt: 

portrait photo of Ailee Cortez James, smiling

This technique works similarly to combining celebrity names except here, you’re using a made up name in your prompt to get a completely unique face. 

I find this method works better than combining celebrity names as it tends to generate a more unique face that isn’t recognizable easily. 

And it also works well with various kinds of prompts. Here’s another example of using the name “Ailee Cortez James” in the prompt:

Prompt:

portrait photo of Ailee Cortez James, wearing a formal suit, inside an office, closeup

Just like celebrity names, if your prompts get too complicated and the subject isn’t close up to the camera, the face will not come out looking the same. 

Before we conclude, there’s another nifty trick to generate consistent faces in Stable Diffusion using random names. 

Instead of writing an actual name in your prompt, you can also use a number in your prompt to get a unique face. Then, you can use that number in your prompt to get consistent faces. 

Here’s an example prompt: 

photo of [56321:12345:0.5], 23-year-old man, smiling

The output image comes looking like this: 

Now, if you use the same number “56321:12345” in another prompt, you can get a similar face.

Prompt: 

photo of [56321:12345:0.5], 23-year-old man, sitting in a park, wearing a jacket

This technique is pretty good and lets you use any random number instead of coming up with unique names for generating faces. 

One thing you need to remember about using random names or numbers for generating consistent faces is that you’ll have to use the same checkpoint models always. 

If you try to generate similar faces with different checkpoint models, you’ll not get consistent faces because each model has different training and this could result in variations. 

Conclusion 

Generating consistent faces in Stable Diffusion is a bit tricky but it isn’t impossible. With the different methods shared in this guide, you’ll be able to get the same face in your images. 

All the methods in this guide are very useful based on your requirements. 

If you want to create a couple of pics using the same face, you can use the method of combining celebrity names or a random name. 

But if you want to create a character and plan on making a lot of images using one face, then you should definitely use ControlNet IP Adapter models or the ReActor extension. 

Either way, you’ll be able to generate stunning consistent faces in Stable Diffusion. Lastly, if you have any questions about any of the methods shared above, feel free to ask them in the comments section below.

Stable Diffusion Prompt Organizer

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.