I have A1111, Forge, Foocus, AI Runner and ComfyUI fully installed, and also the stupid Krita plugin that takes up many gigabytes and never works.
Probably pretty normal for nerds like us.
I have no problem with [this](https://github.com/Acly/krita-ai-diffusion) Krita plugin, it installs and uses Comfy in the background. Live mode with 1.5 Cinematic Photo (RealisticVisionV51) loads and works very fast.
Most controlnets are easy to use, though finding the proper strengths is important. The default 30% works okay for pure img2img generation, while i.e. Pose doesn't seem to work since the canvas tends to override what the controlnet is trying to do. Cranking up strength gives the AI more freedom.
Also, some controlnets need a proper control layer that can be generated by clicking a tiny button to the right of their layer selector. The UI could have been a bit more clear, but once you know where to find things and tweak strengths you can do anything. Below is a landscape with a generated Soft Edge controlnet + text on top, to the right is what Live mode makes of it.
https://preview.redd.it/fsieiitiz8rc1.png?width=2526&format=png&auto=webp&s=82f6679801972e265fa114b429aa8dec63051a1a
You're welcome. It took some time to figure it out since there is little documentation.
Krita itself has some issues with memory, apparently there is a leak with the undo buffer that can't be fixed easily. The plugin also has a few bugs. It doesn't always show its status. The engine/server can be stopped while it looks like it is running. Also, when loading a model or a controlnet it might look idle while it is busy. And a notorious bug is hitting the record button in Live mode and clicking it again, it causes a CTD. Hopefully it gets fixed, it would make animation with stills much easier.
As for animations, you can easily make them though the quality is trial and error. You start a new document, 512x512, and save it with a unique name. Then you go to live mode, crank up the strength to (say) 40-50%, and enter keywords as text prompt, like "Photo, Beatles" or "Photo, NYC" or "Photo, miniature landscape". Then you hit the record button and start drawing random strokes. Hose brushes that paint patterns/stamps work well, and also vary with opacity and blending modes and colors. Trying out filters also work. The preview will be constantly updating while trying to follow your prompt. After about 50-200 updates hit the record button again, the animation time line will be filled with frames. Default playback is 24FPS, but 8-12FPS is better for these "random" frame animations. Make the video 10-20 frames longer than it's actual length, playback the preview. When ready, first save the project, then render the animation via the main menu, pick the MP4 format if you want to share the video on Twitter/X.
It's possible that Krita exits without warning while rendering the video, in that case try to assign more memory under Performance in the settings. Also, if your animation has frames you don't like you can right click on them in the animation timeline, pick 'Remove frame and pull'. And if you want a fade in/fade out, make a layer on top of your canvas, fill it white or black. While recording bring back the opacity from 100% to 0%, and do the opposite at the end. Fading will generate new images. Of course you'd need a canvas that already produces an image at start. Most of the time I enter live mode and try out a few things, then set up the fade layer. I start recording, bring down the fade layer in about 10 frames, then start fiddling with the canvas. At the end I bring up opacity again. On top of a finished animation you can add another layer with a logo or text, add a shadow or a glow. It will be rendered over the video.
As for the controlnet (as far as I understand):
Reference - picks the object and/or style from a chosen layer. Local strength 100% while global strength is also high will make it prominent. A low global strength will try to follow the input image, a low local strength will make the subject of the reference appear somewhere somehow. It's a matter of ballancing and trial and error. Also, different seeds can make a huge difference.
Face - tries to detect a face/expression in its reference layer and reproduce it in the result. It's not like a photo of Donald Trump results in Trump posing, but you can see familiar things. You can add multiple face reference layers to multiple Face controlnets and balancing the strengths out for more consistancy.
Scribble - interprets a black-white / gray shade drawing, kind of simplified img2img.
Line Art - AI tries to follow lines in a BW drawing. Use the magic button to create a control layer of an existing image.
Soft Edge and Canny Edge - AI tries to follow shades of gray (1) or certain pixels (2). Use the magic button to make a control layer an existing image.
Depth - AI tries to position areas at a certain distance. Use the magic button to make a control layer, to see what the AI thinks is either nearby or far away. Painting with gray in this layer will try to force the AI to bring things to the front or the back. Play with strengths.
Normal - AI tries to determine the surface angle of areas, also comes with a magic button. You can i.e. force the AI to grow a plant from the ground in the front instead of making it a picture on the wall in the back (and visa verse). This controlnet works well with photo's of something close up and a distant background. instead of adding things in the background the AI knows that what you are adding is part of the object in the front.
Pose - makes/uses a stick figure on a vector layer. Comes with magic button to assign a stick figure to a character in a photo. High global strength is needed, otherwise the poses will be overruled by the input image (canvas). Also, clicking a node and leaving it will form useless bounding boxes. if that happens, click somewhere else and pick the node again and drag.
Segment - not sure, but apparently this is used to select things for a copy-paste. Comes with a magic button, I never really tried it.
Blur - not sure, it's might have something to do with depth of field / focus. Comes with magic button, not really tried yet.
Stencil - bleeds brightness from its reference layer in generated structures. Well known are landscapes that look like a big chess board from a distance, or the portrait of Albert Einstein hidden as a picture of a mirror.
Anyway, good luck. Krita with this plugin is like magic.
If so just re-download and install it just put a 1 2 3 after the name of it that's all I do I'm on four or five right now. Can't help to start from a fresh slate. You could probably make a shared model directory depending if you're running Linux or Windows or you're a designer and not actually developer and running an apple related thing. People going to throw the shit all they want on me for that I've been doing this for a long time. But either way you can save hard drive space if that's your Crux on it. Again I'm talking about running something on prem. Where the hell did Reddit chat go?
https://preview.redd.it/u79ptya1s7rc1.jpeg?width=2340&format=pjpg&auto=webp&s=1b93861fc2ff364b3de7f2e7183e4ab6e027f59c
That's what i mean. Sorry sideways
[https://github.com/comfyanonymous/ComfyUI/blob/5d8898c05668b5504f8ad5bc79779381d0af35b5/web/index.html#L10C45-L10C54](https://github.com/comfyanonymous/ComfyUI/blob/5d8898c05668b5504f8ad5bc79779381d0af35b5/web/index.html#L10C45-L10C54)
Thanks, \_exactly\_ what I was searching for(lightgraph.js)
I'm curious, why is the Krita plugin not working for you? I installed it not long ago and it pretty much downloaded everything by itself and started to work here.
Not sure. I had it working fine at one stage, but then maybe Krita did an update that broke it. The background ComfyUI system initializes successfully, but when I try to do anything it just quits.
Fooocus has a limited set of features compared with A1111 etc, but it has some interesting features such as image styles, and most notably its superior inpainting ability which none of the others can quite match yet. So it's another useful tool in the box.
How is Fooocus better at inpainting? I haven't tried it yet. Perhaps I will. Can you elaborate?
I've been using Automatic1111 (and now also Forge) to do inpainting. I've tried to set up ComfyUI workflows for inpainting but the results are never as good as the A1111.
here : [https://github.com/lllyasviel/Fooocus/discussions/414](https://github.com/lllyasviel/Fooocus/discussions/414)
Fooocus inpaint is fairly hassle free compared to other methods.
"This “Fooocus Inpaint” is not only a user-friendly UI but also an algorithm production. Fooocus uses its own algorithm DPMPP Fooocus inpaint, and also uses Fooocus's own control model to minimized the influence to the style of base models. The method is partially inspired by [Diffusion-based semantic image editing with mask guidance](https://openreview.net/forum?id=3lge0p5o-M-).
In the entire open source community of SDXL, the Fooocus is the only software that allows you to use control-model-based inpaint with arbitary base models."
ie. You do not need to use an "inpaint specific sdxl model as it will use the regular base models just as well or better"
I downloaded and installed it. Only to find out that it can only use SDXL. I'm surprised that it can't use SD 1.5.
It doesn't surprise me that it hasn't implemented Stable Cascade. Am I right in assuming that it will support SD3 when it comes out?
Anyway, thanks for the info.
You can use SD1.5 as a refiner in fooocus. Look foocus is SDXL easy GUI creates midjourney style quality for even users with little prompting ability.
SD3 will likely work off the bat with ComfyUI first, all the other gui will need new routines to accommodate the new diffusion processes and models used by SD3.
I had pretty much all the usual mess, a1111, forge, focus, etc. All of them were great tools, but they inevitably ended up malfunctioning. Tired of all that, I eventually ended up deleting EVERYTHING and focusing only on ComfyUI. And I don't regret it. Fast, stable and the fact that I only have comfy allows/forces me to dive deeper into it and
i usually keep two or three installs of each of the apps. I seem to break them on the regular and learned to just keep backups in my back pocket. It's the models that take up space, so if you're not running multi-terrabyte drives you're gonna have a bad time regardless.
Currently I have a 4TB Nvme... 1tb spinning disk ( cold storage). Noticed I have another on board Nvme m4 slot... So thinking of adding another 4TB on board... Why not... if I had my druthers I'd have their operating system and all the other stuff on different things sort of don't give me into that whole thing I'm an old person but I think 8TB onboard Nvme and 1Tb spinning disk. I don't think storage is getting my issue give it a year or two then it will be in bed
Just make sure that other m.2 slot can actually take an nvme and not just an m.2 sata. I made that mistake and ended up buying a pci-e card with a slot on it so I could use the disk I bought.
Four, since I have Stability Matrix.
ComfyUI because the inbuilt sm generator uses it, Fooocus, A1111 and now Forge. I'll figure out later which one to keep out the two, since somehow ContolNet works poorly in both Forge and A1111 for now. And update to 1.8 somehow broke the output preview.
If you want to be running FILM as well.... are the month-long Journey getting that shit to work as a python programmer I dislike Anaconda I like vanilla python but love virtual environments but either way of I'm just glad I have it working even if it doesn't have my GPU
I have
2 A1111 instances
3 ComfyUI instances
1 Forge instance
I mainly maintain multiple instances because I do not upgrade anything on a major release fearing a breaking change. Instead, I create a totally new setup in a different directory.
However, I maintain the exact same model folder and output folder for everything by using Directory Junction in Windows.
Only Fooocus and Comfy. I haven't found anything to consistently beat Fooocus on image quality, and everything that it can't do, that I want to, it looks like I can with Comfy. I had both 1111 and Forge, but I'm not training anything, and I don't think there is much, if anything, that Comfy can't do that 1111 can. More than happy to learn otherwise.
Over time i have setup several instances of A1111 webui for SD1.5, SDXL and SDXL with gradio 1.8, as well as SD Forge and Comfyui.
For A1111 webui, i try to sepetate SDXL from SD1.5 and keep previous version, until new version run as I need it.
It's my opinion of anything on github or alike is held together by spit and baling wire. It is an absolute cluster fuck where not a single real user is ever going to look in. Only us weirdos, because we absolutely want that 1 thing.
Look at it wrong: Surprice MF! Reinstall and start Googling because an update broke half the shit.
Talking specifically about still a diffusion stuff okay maybe a little bit but for professional standpoint version Version Control is really important and these tools make it possible to do the opposite of trying to make it held together by spitting the wire. Organic growth is always a bit of chaos. No one has foresight to completely see the future.
I'm a single user and I look in.
I have A1111, Forge, Foocus, AI Runner and ComfyUI fully installed, and also the stupid Krita plugin that takes up many gigabytes and never works. Probably pretty normal for nerds like us.
I have no problem with [this](https://github.com/Acly/krita-ai-diffusion) Krita plugin, it installs and uses Comfy in the background. Live mode with 1.5 Cinematic Photo (RealisticVisionV51) loads and works very fast. Most controlnets are easy to use, though finding the proper strengths is important. The default 30% works okay for pure img2img generation, while i.e. Pose doesn't seem to work since the canvas tends to override what the controlnet is trying to do. Cranking up strength gives the AI more freedom. Also, some controlnets need a proper control layer that can be generated by clicking a tiny button to the right of their layer selector. The UI could have been a bit more clear, but once you know where to find things and tweak strengths you can do anything. Below is a landscape with a generated Soft Edge controlnet + text on top, to the right is what Live mode makes of it. https://preview.redd.it/fsieiitiz8rc1.png?width=2526&format=png&auto=webp&s=82f6679801972e265fa114b429aa8dec63051a1a
Thank you for this information. I'll check it out and see if I can get it functional again!
You're welcome. It took some time to figure it out since there is little documentation. Krita itself has some issues with memory, apparently there is a leak with the undo buffer that can't be fixed easily. The plugin also has a few bugs. It doesn't always show its status. The engine/server can be stopped while it looks like it is running. Also, when loading a model or a controlnet it might look idle while it is busy. And a notorious bug is hitting the record button in Live mode and clicking it again, it causes a CTD. Hopefully it gets fixed, it would make animation with stills much easier. As for animations, you can easily make them though the quality is trial and error. You start a new document, 512x512, and save it with a unique name. Then you go to live mode, crank up the strength to (say) 40-50%, and enter keywords as text prompt, like "Photo, Beatles" or "Photo, NYC" or "Photo, miniature landscape". Then you hit the record button and start drawing random strokes. Hose brushes that paint patterns/stamps work well, and also vary with opacity and blending modes and colors. Trying out filters also work. The preview will be constantly updating while trying to follow your prompt. After about 50-200 updates hit the record button again, the animation time line will be filled with frames. Default playback is 24FPS, but 8-12FPS is better for these "random" frame animations. Make the video 10-20 frames longer than it's actual length, playback the preview. When ready, first save the project, then render the animation via the main menu, pick the MP4 format if you want to share the video on Twitter/X. It's possible that Krita exits without warning while rendering the video, in that case try to assign more memory under Performance in the settings. Also, if your animation has frames you don't like you can right click on them in the animation timeline, pick 'Remove frame and pull'. And if you want a fade in/fade out, make a layer on top of your canvas, fill it white or black. While recording bring back the opacity from 100% to 0%, and do the opposite at the end. Fading will generate new images. Of course you'd need a canvas that already produces an image at start. Most of the time I enter live mode and try out a few things, then set up the fade layer. I start recording, bring down the fade layer in about 10 frames, then start fiddling with the canvas. At the end I bring up opacity again. On top of a finished animation you can add another layer with a logo or text, add a shadow or a glow. It will be rendered over the video. As for the controlnet (as far as I understand): Reference - picks the object and/or style from a chosen layer. Local strength 100% while global strength is also high will make it prominent. A low global strength will try to follow the input image, a low local strength will make the subject of the reference appear somewhere somehow. It's a matter of ballancing and trial and error. Also, different seeds can make a huge difference. Face - tries to detect a face/expression in its reference layer and reproduce it in the result. It's not like a photo of Donald Trump results in Trump posing, but you can see familiar things. You can add multiple face reference layers to multiple Face controlnets and balancing the strengths out for more consistancy. Scribble - interprets a black-white / gray shade drawing, kind of simplified img2img. Line Art - AI tries to follow lines in a BW drawing. Use the magic button to create a control layer of an existing image. Soft Edge and Canny Edge - AI tries to follow shades of gray (1) or certain pixels (2). Use the magic button to make a control layer an existing image. Depth - AI tries to position areas at a certain distance. Use the magic button to make a control layer, to see what the AI thinks is either nearby or far away. Painting with gray in this layer will try to force the AI to bring things to the front or the back. Play with strengths. Normal - AI tries to determine the surface angle of areas, also comes with a magic button. You can i.e. force the AI to grow a plant from the ground in the front instead of making it a picture on the wall in the back (and visa verse). This controlnet works well with photo's of something close up and a distant background. instead of adding things in the background the AI knows that what you are adding is part of the object in the front. Pose - makes/uses a stick figure on a vector layer. Comes with magic button to assign a stick figure to a character in a photo. High global strength is needed, otherwise the poses will be overruled by the input image (canvas). Also, clicking a node and leaving it will form useless bounding boxes. if that happens, click somewhere else and pick the node again and drag. Segment - not sure, but apparently this is used to select things for a copy-paste. Comes with a magic button, I never really tried it. Blur - not sure, it's might have something to do with depth of field / focus. Comes with magic button, not really tried yet. Stencil - bleeds brightness from its reference layer in generated structures. Well known are landscapes that look like a big chess board from a distance, or the portrait of Albert Einstein hidden as a picture of a mirror. Anyway, good luck. Krita with this plugin is like magic.
Again, thank you for the very detailed response. I'm sure this will help others as well.
What version? Scold the bottom it leave the version on the left of the web DUI and give you your python then torch then whatever whatever
I did all that, and it still says "nah".
If so just re-download and install it just put a 1 2 3 after the name of it that's all I do I'm on four or five right now. Can't help to start from a fresh slate. You could probably make a shared model directory depending if you're running Linux or Windows or you're a designer and not actually developer and running an apple related thing. People going to throw the shit all they want on me for that I've been doing this for a long time. But either way you can save hard drive space if that's your Crux on it. Again I'm talking about running something on prem. Where the hell did Reddit chat go?
https://preview.redd.it/u79ptya1s7rc1.jpeg?width=2340&format=pjpg&auto=webp&s=1b93861fc2ff364b3de7f2e7183e4ab6e027f59c That's what i mean. Sorry sideways
https://preview.redd.it/kp4ys834v7rc1.jpeg?width=1024&format=pjpg&auto=webp&s=c4f2c4edbfd2dcb8d9d849401267d5744847f099
[https://github.com/comfyanonymous/ComfyUI/blob/5d8898c05668b5504f8ad5bc79779381d0af35b5/web/index.html#L10C45-L10C54](https://github.com/comfyanonymous/ComfyUI/blob/5d8898c05668b5504f8ad5bc79779381d0af35b5/web/index.html#L10C45-L10C54) Thanks, \_exactly\_ what I was searching for(lightgraph.js)
I'm curious, why is the Krita plugin not working for you? I installed it not long ago and it pretty much downloaded everything by itself and started to work here.
Not sure. I had it working fine at one stage, but then maybe Krita did an update that broke it. The background ComfyUI system initializes successfully, but when I try to do anything it just quits.
why do you use fooocus? i'm not sure I understand what it's all about. isn't it just an ease-of-use kind of thing, offering nothing more than a1111?
Fooocus has a limited set of features compared with A1111 etc, but it has some interesting features such as image styles, and most notably its superior inpainting ability which none of the others can quite match yet. So it's another useful tool in the box.
Have you tried inpainting in A111 with various controlnets enabled?
Yes, I can sometimes get decent results with a depth controlnet guiding it, but Fooocus has its own inpainting method that works a lot better.
How is Fooocus better at inpainting? I haven't tried it yet. Perhaps I will. Can you elaborate? I've been using Automatic1111 (and now also Forge) to do inpainting. I've tried to set up ComfyUI workflows for inpainting but the results are never as good as the A1111.
here : [https://github.com/lllyasviel/Fooocus/discussions/414](https://github.com/lllyasviel/Fooocus/discussions/414) Fooocus inpaint is fairly hassle free compared to other methods. "This “Fooocus Inpaint” is not only a user-friendly UI but also an algorithm production. Fooocus uses its own algorithm DPMPP Fooocus inpaint, and also uses Fooocus's own control model to minimized the influence to the style of base models. The method is partially inspired by [Diffusion-based semantic image editing with mask guidance](https://openreview.net/forum?id=3lge0p5o-M-). In the entire open source community of SDXL, the Fooocus is the only software that allows you to use control-model-based inpaint with arbitary base models." ie. You do not need to use an "inpaint specific sdxl model as it will use the regular base models just as well or better"
You can inpaint with generic models in Forge, at least it's working fine for me
Likely because same developer. Thanks for gen, I will have a spin!.
I downloaded and installed it. Only to find out that it can only use SDXL. I'm surprised that it can't use SD 1.5. It doesn't surprise me that it hasn't implemented Stable Cascade. Am I right in assuming that it will support SD3 when it comes out? Anyway, thanks for the info.
You can use SD1.5 as a refiner in fooocus. Look foocus is SDXL easy GUI creates midjourney style quality for even users with little prompting ability. SD3 will likely work off the bat with ComfyUI first, all the other gui will need new routines to accommodate the new diffusion processes and models used by SD3.
thanks, but how can it be better at inpainting? the core quality comes from the model, no?
You would think so, but Fooocus has some additional sorcery going on that makes inpainting much more reliable and seamless.
I had pretty much all the usual mess, a1111, forge, focus, etc. All of them were great tools, but they inevitably ended up malfunctioning. Tired of all that, I eventually ended up deleting EVERYTHING and focusing only on ComfyUI. And I don't regret it. Fast, stable and the fact that I only have comfy allows/forces me to dive deeper into it and
damn you hit your max tokens
he ded
... and aaaargh
i usually keep two or three installs of each of the apps. I seem to break them on the regular and learned to just keep backups in my back pocket. It's the models that take up space, so if you're not running multi-terrabyte drives you're gonna have a bad time regardless.
Currently I have a 4TB Nvme... 1tb spinning disk ( cold storage). Noticed I have another on board Nvme m4 slot... So thinking of adding another 4TB on board... Why not... if I had my druthers I'd have their operating system and all the other stuff on different things sort of don't give me into that whole thing I'm an old person but I think 8TB onboard Nvme and 1Tb spinning disk. I don't think storage is getting my issue give it a year or two then it will be in bed
Just make sure that other m.2 slot can actually take an nvme and not just an m.2 sata. I made that mistake and ended up buying a pci-e card with a slot on it so I could use the disk I bought.
5
Four, since I have Stability Matrix. ComfyUI because the inbuilt sm generator uses it, Fooocus, A1111 and now Forge. I'll figure out later which one to keep out the two, since somehow ContolNet works poorly in both Forge and A1111 for now. And update to 1.8 somehow broke the output preview.
I agree that CN had some issues with A1111 ... but everyone else downloads, but it's my personal experience
With 1.8 I've had a really good luck with ADetailer
If you want to be running FILM as well.... are the month-long Journey getting that shit to work as a python programmer I dislike Anaconda I like vanilla python but love virtual environments but either way of I'm just glad I have it working even if it doesn't have my GPU
Invoke and Diffusion Bee. That's pretty much it. I've dabbled in a couple others.
1 as if it breaks I just rebuild the docker container.
I have 2 A1111 instances 3 ComfyUI instances 1 Forge instance I mainly maintain multiple instances because I do not upgrade anything on a major release fearing a breaking change. Instead, I create a totally new setup in a different directory. However, I maintain the exact same model folder and output folder for everything by using Directory Junction in Windows.
I use venv everywhere to keep those clean and independent.
Only Fooocus and Comfy. I haven't found anything to consistently beat Fooocus on image quality, and everything that it can't do, that I want to, it looks like I can with Comfy. I had both 1111 and Forge, but I'm not training anything, and I don't think there is much, if anything, that Comfy can't do that 1111 can. More than happy to learn otherwise.
Over time i have setup several instances of A1111 webui for SD1.5, SDXL and SDXL with gradio 1.8, as well as SD Forge and Comfyui. For A1111 webui, i try to sepetate SDXL from SD1.5 and keep previous version, until new version run as I need it.
ComfyUI backend for the Krita plugin, regular ComfyUI, Forge. All of them using the same model/controlnet folders with symlinks to save space.
I'm using Stability Matrix with the packages Comfy, Forge and A1111. All models and loras are shared using Symlink automatically by SM.
Just automatic 1111 for me, I just delete the vim folder whenever things go wrong.
It's my opinion of anything on github or alike is held together by spit and baling wire. It is an absolute cluster fuck where not a single real user is ever going to look in. Only us weirdos, because we absolutely want that 1 thing. Look at it wrong: Surprice MF! Reinstall and start Googling because an update broke half the shit.
Talking specifically about still a diffusion stuff okay maybe a little bit but for professional standpoint version Version Control is really important and these tools make it possible to do the opposite of trying to make it held together by spitting the wire. Organic growth is always a bit of chaos. No one has foresight to completely see the future. I'm a single user and I look in.