How to Colorize Photos with AI - Stable Diffusion + ControlNet Tutorial 2023

Introduction

Restoring and colorizing historical photos is a unique art form, and with the advancement of AI, specifically using Stable Diffusion and ControlNet, the process has become more accessible. Although there's a significant amount of manual work and historical research necessary for an accurate colorization, AI can be an excellent starting point, producing remarkable results with the right techniques.

Getting Started with Stable Diffusion

Before diving into the tutorial, it is essential to ensure you have a basic understanding of image editing and have the Automatic 1111 web UI installed for Stable Diffusion. If you haven’t set this up yet, follow the installation guide linked in the description.

Requirements

Nvidia Windows computer - Suitable for running Stable Diffusion.
Stable Diffusion Web UI - Ensure it's installed.
Checkpoints - Use the "Realistic Vision" model instead of the standard 1.5 or 2.0. Download "Realistic Vision version 2.0" from civitai.com and place it in your stable diffusion models folder.

Installing VAE

Download the VAE file.
Place it into the ‘models’ folder under ‘vae’.
Activate the VAE by going to UI settings and adding , SD_vae to the Quick Settings list.
Apply the settings and reload your UI.

Installing ControlNet

Navigate to the Extensions menu in Stable Diffusion.
Click on "Install from URL" and enter the given URL in the description.
Install and activate ControlNet by checking it in the installed tab.

Preparing the Photo

You can utilize various image editing software like Affinity Photo 2 or Photoshop. Here’s how to proceed with the given Civil War photograph:

Crop the Image: Focus on the subject, using an aspect ratio of 512 by 768 for compatibility with Stable Diffusion.
Resize: Set the resized dimensions to 1536 where applicable.
Clean-Up: Use tools like the Clone Brush to remove artifacts.
Denoising: Apply a light denoising filter to smooth out the image.
Adjustments: Use HSL adjustments to remove saturation and add slight contrast to the image.
Export: Save the cleaned image as a PNG.

Utilizing ControlNet in Stable Diffusion

Import Image: Load the cleaned image into ControlNet.
Preprocessing: Set up three different control models.
- Depth Preprocessor: For depth maps.
- Canny Preprocessor: Adjust thresholds (suggested values: low = 30, high = 130).
- Pose Preprocessor: Use the open pose model.

Configuration Settings

Sampling Method: Set to DPM++ 2M Karras.
Sampling Steps: 30 is a good starting point.
Batch Count: Set according to preference.
Prompt Creation: Use the image-to-image tab to get initial prompts through the CLIP interrogator.

Fine-Tuning the Prompt

Modify the prompt to include more specific details related to historical aspects, such as “dark blue uniform with golden buttons.” Negative prompts can help steer the model away from undesirable outputs such as cartoonish features or undesirable colors.

Generating Results

Generate the first round of images.
Save and analyze the depth, canny, and open pose maps.
Adjust threshold settings or prompts as necessary based on initial results.

Enhancing the Image

The next steps may involve using a combination of image editing techniques:

Color Blending: Use blending modes in your image editor to combine AI-generated color images with the original.
Liquify Tool: Adjust facial features to bring them closer to the original photograph.
Further Color Adjustments: Use curves and saturation adjustments to match the original photo.

Finalizing the Image

Utilize inpainting features to refine specific areas such as facial features. This process allows you to fine-tune the results until you reach a satisfactory level of fidelity to the original.

Conclusion

Colorizing and restoring historical images with AI is a dynamic process that combines manual editing with AI enhancements. While the technology provides a solid foundation, human oversight remains crucial to achieving the desired results. Experimentation and historical knowledge can lead to more accurate and visually appealing colorizations.

Keywords

Colorizing
Historical photos
Stable Diffusion
ControlNet
Image editing
AI techniques

FAQ

Q1: What software is needed for this tutorial?
A1: You need an Nvidia Windows computer and the Stable Diffusion Web UI, alongside an image editing software such as Affinity Photo or Photoshop.

Q2: Can ControlNet be used for other types of images aside from historical photos?
A2: Yes, ControlNet is versatile and can be applied to various types of images needing enhancement or colorization.

Q3: How important is historical research in the colorization process?
A3: Historical research is quite important for accurate colorization, allowing you to make informed choices about the colors and uniforms used in historical contexts.

Q4: Is there a specific input size that works best with Stable Diffusion?
A4: While various sizes can work, ratios like 512 by 768 are recommended for best compatibility with the AI model.

Q5: Do I need advanced photo editing skills to succeed with this process?
A5: Not necessarily! Basic knowledge of image editing software is beneficial, but with practice, anyone can learn to use these techniques effectively.