ComfyUI For Stable Diffusion: The Definitive Guide

Do you wish to create images in Stable Diffusion with full raw power and control? 

If you’re looking for a Stable Diffusion web UI that is designed for advanced users who want to create complex workflows, then you should probably get to know more about ComfyUI. 

In this comprehensive guide, I’ll cover everything about ComfyUI so that you can level up your game in Stable Diffusion. 

Here’s what I’ll cover: 

• What’s ComfyUI and how it works
• How to install ComfyUI 
• Best ComfyUI workflows to use
• Best extensions to be more fast & efficient 

By the end of this ComfyUI guide, you’ll know everything about this powerful tool and how to use it to create images in Stable Diffusion faster and with more control. 

Let’s jump right in. 

What is ComfyUI & How Does it Work? 

ComfyUI is a node-based interface to use Stable Diffusion which was created by comfyanonymous in 2023. 

Unlike other Stable Diffusion tools that have basic text fields where you enter values and information for generating an image, a node-based interface is different in the sense that you’d have to create nodes to build a workflow to generate images. 

While this may all seem a bit new or unique to you, node-based interfaces are pretty common for creative work and are used in popular tools such as Blender, Cinema 4D, Maya, Davinci Resolve, Unreal Engine, and more. 

In fact, node-based tools are a standard for these industries as they give more control and freedom over the workflow. 

If you don’t understand how these node-based tools work, it’s pretty simple. Every node is intended to execute a code. 

Nodes are able to take inputs as well as provide outputs. In ComfyUI, you’ll use nodes to: 

• provide inputs such as checkpoint models, prompts, images, etc
• Modify or edit parameters of nodes such as sample steps, seed, CFG scale, etc 
• Get output from nodes in the form of images

Nodes can be easily created and managed in ComfyUI using your mouse pointer. 

In ComfyUI, there are nodes that cover every aspect of image creation in Stable Diffusion. So, you’ll find nodes to load a checkpoint model, take prompt inputs, save the output image, and more. 

By combining various nodes in ComfyUI, you can create a workflow for generating images in Stable Diffusion. 

ComfyUI vs Automatic1111

You might be wondering why go through all this hassle to create images when you can do so easily using Automatic1111 which is by far the most popular and commonly used interface for Stable Diffusion. 

Not only this, the image of ComfyUI shown above looks so overwhelming that you might be not even willing to go through all that trouble to use it. 

And that’s fair. 

When I first came across ComfyUI, that was my exact reaction and I thought the same as it made no sense to spend so much time doing something that can be done in seconds in Automatic1111. 

But when you use ComfyUI, you’ll realize how much better it is compared to Automatic1111. In my opinion, it totally obliterates Automatic111 when it comes to power. 

It is so good that the creators of Stable Diffusion over at StabilityAI use ComfyUI in their workflows. 

Here are some key differences between ComfyUI and Automatic1111 to help you understand why people are so crazy about ComfyUI. 

Related: Fooocus Guide: The Fastest Way To Run Stable Diffusion

Node-Based UI 

The key difference between ComfyUI and Automatic1111 is that ComfyUI has a node-based interface whereas Automatic1111 has a typical interface with input fields. 

On first look, Automatic1111 looks very straightforward but it can be very limiting because you can only do what their interface lets you do. 

On the other hand, ComfyUI lets you build your own process for using Stable Diffusion. This means you can do so much more using the various nodes available in ComfyUI. 

You can run multiple generations at once, output preview images at different stages of generation, and more. 

More Streamlined Workflow

ComfyUI has a more streamlined workflow compared to Automatic1111. As mentioned above, you can only do things as the Automatic1111 interface allows you to do. 

But since you can build your own workflows in ComfyUI, you are not restrained in any manner. 

You might be wondering how is this beneficial as a user. 

Well, since you can build your own workflows, you can combine different processes or stages of image generation in one workflow. 

Here are some ways ComfyUI is better in terms of workflow compared to Automatic1111: 

• Run a txt2img with a latent upscale or hi-res fix in one step automatically in a single click.
• Run txt2img and img2img in one workflow in a single click. 
• Use different models for base sampler and refiner in one generation. 

These examples alone should be enough to understand how powerful ComfyUI actually is. You can build any workflow imaginable using it and you can do it all in one generation without having to switch tabs and click generate multiple times. 

As a bonus, you can find so many great ComfyUI workflows made by the community that will help you be more efficient in generating images. 

Faster Performance

ComfyUI is better optimized to run Stable Diffusion compared to Automatic1111.

Many users on the Stable Diffusion subreddit have pointed out that their image generation times have significantly improved after switching to ComfyUI. 

Moreover, SDXL works much better in ComfyUI as the workflow allows you to use the base and refiner model in one step. This also adds to the overall performance as you save time by not switching to img2img to run the refiner. 

Since ComfyUI has better performance and is optimized well, it also helps people using less powerful GPU or low vRAM run Stable Diffusion effectively. 

For reference, I have a laptop with an Nvidia RTX 3050 card with 4GB of vRAM and I’m able to use SDXL models in ComfyUI. With Automatic1111, I was never able to do it as I would get memory errors when loading models or during image generation. 

But now, I’m able to generate 4K images with SDXL models in under 4-5 minutes using ComfyUI. That’s impossible for anyone with low vRAM to do it in Automatic1111. 

So, if your system doesn’t optimally meet Stable Diffusion requirements, you’d benefit from using ComfyUI a lot.

Steep Learning Curve

ComfyUI without a doubt has a steep learning curve compared to Automatic1111. 

Automatic1111 is perfect for anyone who wants to create pretty pictures to share with their friends or on social media.  

But if you want anything more than that, it’s worth learning ComfyUI. That’s because you’ll not only be learning a powerful tool, but you’ll also get a deeper understanding of how Stable Diffusion actually works. 

ComfyUI’s node-based interface helps you get a peak behind the curtains and understand each step of image generation in Stable Diffusion. 

So, if you plan on using Stable Diffusion for professional work, you’ll be better off learning and using ComfyUI. 

How To Install ComfyUI 

The process of installing ComfyUI is very simple and straightforward. Here are the different ways to install ComfyUI: 

Quick Portable Install (Windows)

ComfyUI comes with a portable version that can be installed with a single click. If you don’t want to install using a Command Prompt, I’d recommend this method. 

The portable installer lets you use ComfyUI with both CPU and GPU but the CPU generation times are much slower so only use this method if you want to use ComfyUI with your GPU. 

Download Link 

Click on the link above and ComfyUI will be downloaded to your computer. Extract the .zip file to the location where you want to store ComfyUI. I’d recommend you extract it to a drive with atleast 10-20GB of free storage space. 

Once your file is extracted, you’ll see a folder named ComfyUI_windows_portable with various files. 

Double-click on the run_nvidia_gpu.bat file if you want to run ComfyUI with your GPU or run_cpu.bat to run it with your CPU. 

This will open the Command Prompt which starts ComfyUI. Once ComfyUI has started, it’ll automatically open up a window where the ComfyUI interface will be loaded. 

GitHub Clone Using CMD (Windows & Linux) 

For Nvidia GPU 

In Windows, open your Command Prompt by searching “cmd” on the start menu. If you’re on Linux, open your Terminal by using the shortcut key Ctrl+Alt+T. 

In your Command Prompt/Terminal, run the following commands one by one. 

git clone https://github.com/comfyanonymous/ComfyUICopy

cd ComfyUI Copy

pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu118 xformers -r requirements.txtCopy

This will install ComfyUI on your device and now you’ll be able to launch it using the following command: 

python main.pyCopy

There could be cases where you get an error “Torch not compiled with CUDA enabled”. If that happens, just uninstall Torch using the following command: 

pip uninstall torchCopy

Then reinstall it using this command: 

pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu118 xformers -r requirements.txtCopy

For AMD GPU 

In Windows, open your Command Prompt by searching “cmd” on the start menu. If you’re on Linux, open your Terminal by using the shortcut key Ctrl+Alt+T. 

In your Command Prompt/Terminal, run the following commands one by one. 

git clone https://github.com/comfyanonymous/ComfyUICopy

cd ComfyUICopy

python -m pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/rocm5.4.2 -r requirements.txtCopy

This will install ComfyUI on your device and now you’ll be able to launch it using the following command: 

python main.pyCopy

GitHub Clone Using CMD (Mac) 

If you have an M1 or M2 Mac, you install ComfyUI. First, you’ll have to install PyTorch by following this guide from Apple. 

Open your Terminal and clone the ComfyUI repository by using this command: 

git clone https://github.com/comfyanonymous/ComfyUICopy

Now, install the ComfyUI dependencies using these commands: 

cd ComfyUICopy

pip install -r requirements.txtCopy

This will install ComfyUI on your Mac and now you’ll be able to run it with the following command: 

python main.py --force-fp16Copy

Related: One-Click Install ComfyUI on Mac

How To Update ComfyUI 

Updating ComfyUI is pretty simple as well. Here’s how you can do it:

For Portable Version 

If you’ve installed ComfyUI using the quick install portable version, you can update it in a single click without using the Command Prompt. 

Navigate to the folder where you’ve installed ComfyUI. Here go to ComfyUI > Update folder. 

In this folder, double-click on the update_comfyui.bat and it’ll automatically update ComfyUI for you. 

If you also want to update the Python dependencies along with ComfyUI, double-click on the update_comfyui_and_python_dependencies.bat file instead. 

For GitHub Version 

If you’ve installed ComfyUI using GitHub (on Windows/Linux/Mac), you can update it by navigating to the ComfyUI folder and then entering the following command in your Command Prompt/Terminal: 

git pullCopy

How To Use ComfyUI 

To use ComfyUI, the first thing you need to understand is its interface and how nodes work. 

Understanding The ComfyUI Interface

When you open ComfyUI, you’ll either see a blank screen or the default workflow will be loaded as shown below:

If you have a workflow loaded, select all the nodes by clicking the Ctrl+A and delete them by pressing the Delete key on your keyboard. Or click on the Clear button from the sidebar.

We’re doing this because it’s better if you start from scratch and learn every single thing along the way. 

Adding Nodes

As a node-based UI, ComfyUI works entirely using Nodes. 

To add a node, right-click on the blank space mouse and select the Add Node option. Under this, you’ll find the different nodes available. 

Alternatively, you can also add nodes by double-clicking anywhere on the blank space and typing the name of the node you want to add. 

You can select a node by clicking on it and also move it around by dragging the node with your mouse. 

You can also select multiple nodes by holding the Ctrl (Cmd) key while selecting the nodes. Once multiple nodes are selected, you can move them together by holding the Shift key and dragging them around. 

When you right-click on the node, a menu opens up with different options. You can view the properties of the node, remove the node, change the node color, and more. 

You can also resize the nodes by hovering your mouse over the edge at the bottom right and dragging it to change the size. 

Besides the nodes, the ComfyUI interface also has a sidebar on the right where you’ll find the button to execute the prompt along with the following options: 

• Save: Save the current workflow as a .json file
• Load: Load a ComfyUI .json file workflow 
• Refresh: Refresh ComfyUI workflow
• Clear: Clears all the nodes on the screen 
• Load Default: Load the default ComfyUI workflow 

In the above screenshot, you’ll find options that will not be present in your ComfyUI installation. 

That’s because I have installed additional ComfyUI extensions which we’ll get to later in this guide.

Connecting Nodes

To build a workflow for image generation in ComfyUI, you’ll have to connect nodes to each other. This is the step where people get confused or overwhelmed. 

But it’s not that difficult. Let me walk you through it. In the image below, I have a Checkpoint Loader node added which has three node outputs. 

These nodes are supposed to be connected to another node as input. 

Hold and drag the model node and release it on the blank space. This opens a list of compatible nodes you can connect to the Model node. Let’s select the KSampler node here. 

Now, you’ll see the new node has been added and it has multiple input nodes. 

You’ll also notice that each node has different colors. This is for better identification purposes as you cannot connect nodes of different colors.

Notice when I connect the Model node (purple color) to the Positive node (orange), it doesn’t connect.

These colors will help you a ton in understanding what node is supposed to go where. 

Prompt Execution

Click on the Load Default button to load the default ComfyUI workflow. 

This is a basic txt2img workflow that works in a similar way to Automatic1111. It has all the nodes for txt2img generation. 

To execute the prompt, click on the Queue Prompt button. This starts the workflow and you’ll see it pass through each node with a green highlight. 

When the workflow is complete, it ends at the last node and the prompt is completed. 

Since this is the default workflow, it executed perfectly without errors. But there could be instances when you’re building your own workflow and have a node not connected properly or missing. 

In such cases, ComfyUI will show an error message indicating the missing node as shown below. 

In our example, the Load Checkpoint model output node is not connected to the KSampler model input node. ComfyUI also highlights the KSampler node to visually indicate where the error occurred. 

This will help you a lot in understanding ComfyUI and even when you make mistakes or run into errors, ComfyUI will tell you what went wrong exactly. 

ComfyUI Nodes Explained

I hope by now you’re getting a hang of how ComfyUI works. The next thing you’re going to learn is what all these nodes mean and do. 

Here’s the default workflow for txt2img in ComfyUI. I’ll be using this workflow as an example to explain every node used here and what purpose it serves. 

Load Checkpoint Node

This node is used to load a checkpoint model in ComfyUI. You can load a .safetensors or .ckpt checkpoint model in this node. 

Just like you first select a checkpoint model in Automatic1111, this is usually the first node we add in ComfyUI as well. 

The Load Checkpoint node has three output nodes:

• Model (Unet): The model used for the image generation process is fed to the KSampler 
• CLIP: to encode text prompts into a format the sampler can understand 
• VAE: to encode or decode an image to and from latent space to pixel space

CLIP Text Encode Node

The CLIP output from the Load Checkpoint node connects directly to the input of the CLIP Text Encode Node. 

This node is used to encode the text (your prompts) into a format the model (Unet) can understand. 

We also need to have two CLIP Text Encode nodes for positive and negative prompts. 

The CLIP Text Encode Node has one conditioning node as an output which connects to our next node. 

KSampler Node

The KSampler node is the node responsible for the sampling process in Stable Diffusion. In other words, this is the node where the text (prompt) is converted into an image. 

Since this is the sampling node in ComfyUI, it also takes the most time to run during the image generation process. 

The KSampler node has the following input nodes: 

• Model: The model (Unet) output from the Load Checkpoint node
• Positive: A positive conditioning (positive prompt) from the CLIP Text Encode Node 
• Negative: A negative conditioning (negative prompt) from the CLIP Text Encode Node 
• Latent Image: An image in latent space which is an input from the Empty Latent Image node

Empty Latent Image Node

The Empty Latent Image node is used when you want to pass an empty latent image to the sampler. 

This is used for txt2img generation when you just want to specify the image dimensions and generate an image. 

It has a Latent node as an output which connects to the Latent input node of the KSampler. 

For img2img generation, we use a different node in that case as we don’t pass an empty latent image.

VAE Nodes

VAE Nodes are used for either encoding or decoding an image to and from the latent space. 

Since we want to decode our image from latent space to pixel space, we’ll use the VAE Decode node. It has the following input nodes: 

• Samples: The latent image that was generated by the sampler (KSampler)
• VAE: The VAE that came with our checkpoint model 

The VAE Decode node has an Image node as the output node which brings us to our last node. 

Save Image 

The Save Image node is used to save the decoded image in our preferred destination. You can also view the generated image in this node. 

Best ComfyUI Workflows

The default ComfyUI workflow is one of the simplest workflows and can be a good starting point for you to learn and understand ComfyUI better. 

However, there are many other workflows created by users in the Stable Diffusion community that are much better, complex, and powerful. 

Here’s a table listing some of the best ComfyUI workflows along with their purpose and ease of use. 

SD1.5 Template Workflows for ComfyUI

This workflow is intended to work on SD1.5 models and comes with three versions based on its complexity. 

So, you can use the simple, intermediate, or advanced version depending on your knowledge and experience in ComfyUI. 

Simply Comfy

The default ComfyUI workflow doesn’t have a node for loading LORA models. This simple workflow is similar to the default workflow but lets you load two LORA models. 

It works with all models that don’t need a refiner model. So, you can use it with SD1.5 models and SDXL models that don’t need a refiner. 

SDXL Config ComfyUI Fast Generation 

Many users who have a low-powered GPU or less vRAM face difficulties in generating images fast in ComfyUI. The workflow is intended to overcome that problem and generate images in under 2-3 minutes. 

It has the SDXL base and refiner sampling nodes along with image upscaling. This SDXL ComfyUI workflow has many versions including LORA support, Face Fix, etc. 

This is often my go-to workflow whenever I want to generate images in Stable Diffusion using ComfyUI. That’s because the creator of this workflow has the same 4GB RTX 3050 card configuration that I have on my system. 

Searge-SDXL: EVOLVED

I would’ve loved to put this on top of the list since it’s the ultimate ComfyUI workflow but it’s also advanced that many beginners would get overwhelmed. 

This workflow supports txt2img, img2img, inpainting, Controlnet, multiple LORAs, and more. It also has a styling node allowing you to define the image style in your prompt separately. 

SDXL ComfyUI Ultimate Workflow

This is another very powerful comfyUI SDXL workflow that supports txt2img, img2img, inpainting, Controlnet, face restore, multiple LORAs support, and more. 

The workflow also has a prompt styler where you can pick from over 100 Stable Diffusion styles to influence your image generation. 

SDXL ControlNet/Inpaint Workflow

If you want ComfyUI inpainting or Controlnet workflow, this one is definitely a good one for beginners and intermediate users. 

The workflow is pretty straightforward and works with SDXL models. 

Simple SDXL Inpaint

This is one of the simplest and easiest-to-use ComfyUI inpainting workflows that works with SDXL. 

It’s as straightforward as the default ComfyUI workflow where you just select the model, load your image for inpainting, and click generate. 

Sytan’s SDXL Workflow

Another SDXL comfyUI workflow that is easy and fast for generating images. It supports txt2img with a 2048 upscale. 

Best ComfyUI Extensions & Nodes

When you’re using different ComfyUI workflows, you’ll come across errors about certain nodes missing. That’s because many workflows rely on nodes that aren’t installed in ComfyUI by default. 

Besides this, many extensions are available that make ComfyUI much better than it already is. 

Here’s a list of some of the best ComfyUI extensions and nodes you should install: 

• ComfyUI Manager: Allows you to detect and install missing nodes in any workflow and even updates ComfyUI from the UI itself. 
• ComfyUI Impact Pack: Adds additional upscaler, image detector, and detailer nodes to ComfyUI.
• ComfyUI Controlnet Preprocessors: Adds preprocessors nodes to use Controlnet in ComfyUI. 
• WAS Node Suite: A node suite with over 100 nodes for advanced workflows. 

Conclusion

At first, using ComfyUI will seem overwhelming and will require you to invest your time into it. But as you start to get a hang of it, you’ll realize why many users are ditching Automatic1111 for ComfyUI. 

It gives you full freedom to use Stable Diffusion at its raw power by building workflows that are highly advanced and perfect for anyone generating images for professional work. 

I hope this comprehensive ComfyUI guide helped you learn more about it and now you’ll be able to build your own workflows and start generating images in it. 

FAQs

Here are some frequently asked questions about ComfyUI: