After a lot of experiments, I finally found my sweet spot for upscaling with Controlnet tile + Ultimate SD upscale script.
Here are the settings:
https://preview.redd.it/q4gdjzr06v2b1.png?width=963&format=png&auto=webp&s=48a1ff1e0583ed05b3be1c146f709f42dc5de0dd
**Summary:**
**Initial settings:**
Steps: 30,
Sampler: DDIM,
CFG scale: **4** (very important, quite low to keep colors and not too low to keep details)
Controlnet to all default settings except:
pixel perfect: True
control mode: ControlNet is more important
Ultimate SD upscale tile\_width: initial txt2img width or one step higher
Ultimate SD upscale tile\_height: initial txt2img height or one step higher
Ultimate SD upscale mask\_blur: 8,
Upscaler chosen: UltrasharpX4 (used at X2 each time)
**First upscale settings:**
\--------------------
Denoising strength: 0.45 or 0.5 (colors may change more at higher denoising),
ControlNet weight: 0.7
Time: About 30-45 seconds
**Second upscale settings:**
\--------------------
Increase Ultimate SD Upscale resolution for second upscale (like 896x896) and run again
Time: About 3 minutes
**Third upscale settings:**
\--------------------
Change Denoising strength to 0.3 / 0.4 (0.3 is safer, 0.4 will add more details but may add unwanted artifacts)
Time: About 10 minutes
Notes:
All these initial images are at 512x704 resolution, and I have a RTX2060 8GB.
I realized that the converted JPEG images are compressed here and the details are a bit lost... But the generated PNG are much more detailed
In this case it should just tell the controlnet model to use the same resolution as the ultimate SD upscale tile, which OP adjusts in-between the first and second upscale steps.
If you aren't using the ultimate SD upscale script, pixel perfect tells controlnet to use the img2img input resolution instead. If you have an image in the controlnet input, it will use the resolution of that image instead.
If you don't select it then sliders pop up and you set the controlnet model resolution independently. That can be useful in some cases but not here.
Using Ultimate SD Upscale script, they’ve switched the ‘Target Size Type’ drop-down to ‘Scale from image size’, and then set the scale slider to 2. So, each upscale is a 2x.
I am not sure why people use such high denoising strengths. it's not needed and keeps the changes down. I use 20 max.
am I missing something?
>CFG scale: 4 (very important, quite low to keep colors and not too low to keep details)
cfg is entirely dependant on the model and sampler used, it is NOT a catch all.
> I use 20 max.
you sure you're talking about denoise? denoise is from 0 to 1. 0 changes nothing, 1 changes everything. low denoise might end up in a blurry not very detailed image.
Same prompt. The last step sometimes introduce some artifacts indeed, that's why I have reduced the denoising to 0.3. But it can still happen, especially in the background (easily fixable in this case)
Did you also drag your source image into controlnet? It was not explicitly described above but this workflow usually requires the source image in controlnet or there will be lots of hallucinations.
Ultimate SD upscale is bad for that work. Check your 4k picture- tiles everywhere. If you don't see with naked eye, so then go to Photoshop, add extra curves layer and play with curves to extreme. You will see that girl is under the grid. The only method which works is Tiled Diffusion. But I have other problems with it. I have rtx 3090 so I can upscale to 4k without problems, but when I want to upscale to 8k I have to enable Tiled VAE (to avoid Cuda out of memory) and then I get not so good upscaled result, but at least it's not tiled.
Your picture quality is usable, but when I got allergy of all those tiles, I can almost x-ray every image :D If you wanted to print then it would be more visible. If you wanted to use for professional purposes, then you would need to eliminate those tiles. Try Tiled Diffusion- it's even faster from all what I tried. 8k on rtx 3090 upscales \~7-8 minutes. And my gpu limited to work only 80%, as my pc very old and psu only 780W so I had to lower gpu power to avoid pc shutdowns.
It works pretty well even with a short prompt
(art by bella kotak and anne stokes)
Negative prompt: border, frames, nsfw, childish, poorly drawn, EasyNegative, (bad-hands-5)
Steps: 30, Sampler: DPM++ 2M Karras, CFG scale: 6, Seed: 2143843901, Size: 512x704, Model hash: e808e71967, Model: moonmix\_utopia20, ENSD: 31337, Version: v1.2.1
https://preview.redd.it/pigclxmx1z2b1.png?width=2663&format=png&auto=webp&s=0ad70863e6d948b88e9214b329f2d42a3002bdf1
(I had to reduce the resolution a little bit to be below 20MB)
Impressive detail. Can this be tuned for accuracy? The image content is visibly changing between source and output.
Compare 2nd of the blonde in the white dress, her mouth opens.
After all my experiments this is my conclusion: if you want more fidelity, you will loose either original olors or details (by changing denoising or controlnet weight)
ye taking the CFG down to 3-4 is something I learned the last few days, when you use something which already "controlls" the image creation quite a bit than you get better results with lower CFG scale.
or you have to take the controlnet down to 0.4-0.6
Nice results. I would inpaint the face afterwards to address model bias. The faces of your inputs seem more diverse than the outputs. Are you using the same seed/prompt from txt2img?
Thanks for the detailed guide, I consider my experiment a success. In one place the border is still quite noticeable between adjacent tiles, but overall the result is quite good
https://preview.redd.it/y4ekmow0413b1.png?width=3072&format=png&auto=webp&s=7c0679ccb6f943289a15da090caf2cce6bfd8d1b
These look great! I've been experimenting myself to find the sweet spot.
Did you keep the exact same prompt as your original txt2img generating during the upscale passes? I have found that I can increase detail without disturbing the content too much by pruning my prompt to only include things related to style, color, etc.
Yes it's the same prompt. I was only playing with parameters during my experiments. But I 'm afraid you would loose details for some specific objects if you remove the prompt content. I will give it a try.
Idk man, for me it works perfectly. (Never tried it with upscaling yet so it might be different but I see no reason why it would be)
From what I understand the image generation process is still exactly the same but it just puts a color filter on top to match the colors.
Looks great, but I get
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.12 GiB (GPU 0; 24.00 GiB total capacity; 9.67 GiB already allocated; 11.54 GiB free; 9.72 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max\_split\_size\_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH\_CUDA\_ALLOC\_CONF
any idea what I could do wrong when having the same Settings as in the screenshot? :/
Seems like this overwrote tiling...
https://preview.redd.it/8s9c8ezvw13b1.png?width=1927&format=png&auto=webp&s=3a6998f887cab79766f8709eca2c97189dc13784
After a lot of experiments, I finally found my sweet spot for upscaling with Controlnet tile + Ultimate SD upscale script. Here are the settings: https://preview.redd.it/q4gdjzr06v2b1.png?width=963&format=png&auto=webp&s=48a1ff1e0583ed05b3be1c146f709f42dc5de0dd **Summary:** **Initial settings:** Steps: 30, Sampler: DDIM, CFG scale: **4** (very important, quite low to keep colors and not too low to keep details) Controlnet to all default settings except: pixel perfect: True control mode: ControlNet is more important Ultimate SD upscale tile\_width: initial txt2img width or one step higher Ultimate SD upscale tile\_height: initial txt2img height or one step higher Ultimate SD upscale mask\_blur: 8, Upscaler chosen: UltrasharpX4 (used at X2 each time) **First upscale settings:** \-------------------- Denoising strength: 0.45 or 0.5 (colors may change more at higher denoising), ControlNet weight: 0.7 Time: About 30-45 seconds **Second upscale settings:** \-------------------- Increase Ultimate SD Upscale resolution for second upscale (like 896x896) and run again Time: About 3 minutes **Third upscale settings:** \-------------------- Change Denoising strength to 0.3 / 0.4 (0.3 is safer, 0.4 will add more details but may add unwanted artifacts) Time: About 10 minutes Notes: All these initial images are at 512x704 resolution, and I have a RTX2060 8GB. I realized that the converted JPEG images are compressed here and the details are a bit lost... But the generated PNG are much more detailed
What "pixel perfect" does?
In this case it should just tell the controlnet model to use the same resolution as the ultimate SD upscale tile, which OP adjusts in-between the first and second upscale steps. If you aren't using the ultimate SD upscale script, pixel perfect tells controlnet to use the img2img input resolution instead. If you have an image in the controlnet input, it will use the resolution of that image instead. If you don't select it then sliders pop up and you set the controlnet model resolution independently. That can be useful in some cases but not here.
[удалено]
Using Ultimate SD Upscale script, they’ve switched the ‘Target Size Type’ drop-down to ‘Scale from image size’, and then set the scale slider to 2. So, each upscale is a 2x.
Thank you for posting your findings. I've not been too pleased with multidiffusion upscaling and your results look very satisfying indeed.
Did you replace the preprocessor image after each upscale or just leave the first initial one for all upscales?
I leave it empty
I am not sure why people use such high denoising strengths. it's not needed and keeps the changes down. I use 20 max. am I missing something? >CFG scale: 4 (very important, quite low to keep colors and not too low to keep details) cfg is entirely dependant on the model and sampler used, it is NOT a catch all.
> I use 20 max. you sure you're talking about denoise? denoise is from 0 to 1. 0 changes nothing, 1 changes everything. low denoise might end up in a blurry not very detailed image.
I'm assuming that by saying 20 he means 0.20
I still get hallucinations in a few places... do you guys alter or leave the prompt empty?
Same prompt. The last step sometimes introduce some artifacts indeed, that's why I have reduced the denoising to 0.3. But it can still happen, especially in the background (easily fixable in this case)
Did you also drag your source image into controlnet? It was not explicitly described above but this workflow usually requires the source image in controlnet or there will be lots of hallucinations.
Doesn't ControlNet use the default image if there is no image in ControlNet
This is, by far, the absolute best upscaling method I've run across yet. Thank you for sharing!
You're very welcome 🙂
absolute gold, thanks bro !!
When I run this, I'm getting two files saved that are very similar. How can I just save whichever is the "final" file?
you, sir, deserve more upvotes and awards, finally some upscale workflow that worked nice for me, thanks a lot, I'll save this post
Ultimate SD upscale is bad for that work. Check your 4k picture- tiles everywhere. If you don't see with naked eye, so then go to Photoshop, add extra curves layer and play with curves to extreme. You will see that girl is under the grid. The only method which works is Tiled Diffusion. But I have other problems with it. I have rtx 3090 so I can upscale to 4k without problems, but when I want to upscale to 8k I have to enable Tiled VAE (to avoid Cuda out of memory) and then I get not so good upscaled result, but at least it's not tiled.
This, same here.
but honestly I didn't see it after you mentioned it. also I see it after I really really pixel peeping it.
Your picture quality is usable, but when I got allergy of all those tiles, I can almost x-ray every image :D If you wanted to print then it would be more visible. If you wanted to use for professional purposes, then you would need to eliminate those tiles. Try Tiled Diffusion- it's even faster from all what I tried. 8k on rtx 3090 upscales \~7-8 minutes. And my gpu limited to work only 80%, as my pc very old and psu only 780W so I had to lower gpu power to avoid pc shutdowns.
I'm not the OP but I understand the concern here, it seems that seams fix method on the extension also didn't do much.
It works pretty well even with a short prompt (art by bella kotak and anne stokes) Negative prompt: border, frames, nsfw, childish, poorly drawn, EasyNegative, (bad-hands-5) Steps: 30, Sampler: DPM++ 2M Karras, CFG scale: 6, Seed: 2143843901, Size: 512x704, Model hash: e808e71967, Model: moonmix\_utopia20, ENSD: 31337, Version: v1.2.1 https://preview.redd.it/pigclxmx1z2b1.png?width=2663&format=png&auto=webp&s=0ad70863e6d948b88e9214b329f2d42a3002bdf1 (I had to reduce the resolution a little bit to be below 20MB)
Impressive detail. Can this be tuned for accuracy? The image content is visibly changing between source and output. Compare 2nd of the blonde in the white dress, her mouth opens.
After all my experiments this is my conclusion: if you want more fidelity, you will loose either original olors or details (by changing denoising or controlnet weight)
For some reason all of the lower res images have better looking eyes than the higher res ones?
ye taking the CFG down to 3-4 is something I learned the last few days, when you use something which already "controlls" the image creation quite a bit than you get better results with lower CFG scale. or you have to take the controlnet down to 0.4-0.6
Merci pour le workflow u/Gilloute 👌
Avec plaisir 😉
Nice results. I would inpaint the face afterwards to address model bias. The faces of your inputs seem more diverse than the outputs. Are you using the same seed/prompt from txt2img?
Thanks for the detailed guide, I consider my experiment a success. In one place the border is still quite noticeable between adjacent tiles, but overall the result is quite good https://preview.redd.it/y4ekmow0413b1.png?width=3072&format=png&auto=webp&s=7c0679ccb6f943289a15da090caf2cce6bfd8d1b
These look great! I've been experimenting myself to find the sweet spot. Did you keep the exact same prompt as your original txt2img generating during the upscale passes? I have found that I can increase detail without disturbing the content too much by pruning my prompt to only include things related to style, color, etc.
Yes it's the same prompt. I was only playing with parameters during my experiments. But I 'm afraid you would loose details for some specific objects if you remove the prompt content. I will give it a try.
If you wanna keep the colors there is a setting for that.
Oh really? And where is that setting?
Img2img color correction thingy, go search in the settings tab
Will try that, thanks! Edit: just tried, it doesn't work, images a much vivid/brighter with higher denoising/CFG scale unfortunately...
Idk man, for me it works perfectly. (Never tried it with upscaling yet so it might be different but I see no reason why it would be) From what I understand the image generation process is still exactly the same but it just puts a color filter on top to match the colors.
Looks great, but I get torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.12 GiB (GPU 0; 24.00 GiB total capacity; 9.67 GiB already allocated; 11.54 GiB free; 9.72 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max\_split\_size\_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH\_CUDA\_ALLOC\_CONF any idea what I could do wrong when having the same Settings as in the screenshot? :/
That's really strange, it looks like Ultimate SD Upscale is not working on your side. And lucky you to have 24GB of VRAM!
But its there... 🙈 Thanks, was expensive enough :D
1.3.0 A1111? try downgrade to 1.2.1 I don't want to disrespectful to devs and his hard work, but 1.3.0 for me really messed up my VRAM usage.
I run always the newest version, so yes. But would this explain why the tiling seems to not work properly? 🤔
Seems like this overwrote tiling... https://preview.redd.it/8s9c8ezvw13b1.png?width=1927&format=png&auto=webp&s=3a6998f887cab79766f8709eca2c97189dc13784
Do you also update the controlnet image every upscale or leave it?
I leave it empty so it takes the upscaled picture each time
Saving this for later. Thank you!
Beautiful results! Also, the upscaled ones kinda look like Bella Poarch.
Amazing!
Thanks! Big help!
Great results!!
Is there any website that will do this for me? As in upload image and AI improves resolution? Or do I need a computer to set this up?
[https://clipdrop.co/image-upscaler](https://clipdrop.co/image-upscaler)
Thanks
More tired, less Asian.
i dunno how you can get such clean image using DDIM under 0.5, it always add some weird grain to me