[ControlNet 1.1] The updating track.

[lllyasviel]

We will use this repo to track some discussions for updating to ControlNet 1.1.

[lllyasviel]

Update: ControlNet 1.1 is released here.

[lllyasviel]

I think we can ignore cnet11 Tile model right now. We are not very sure how to make use of it.
The inpainting model may need more considerations in implementation and perhaps we just get other models first.

[continue-revolution]

The inpainting model may need more considerations in implementation and perhaps we just get other models first.

I’m the author of sd-webui-segment-anything and I am planning to connect my extension to your inpainting model.

So at this moment, the inpainting ControlNet cannot target at the mask only while not changing other parts, right?

Edit on 2023/04/18 already connected. Checkout my extension readme for how to use.

[pkuliyi2015]

I think we can ignore cnet11 Tile model right now. We are not very sure how to make use of it. The inpainting model may need more considerations in implementation and perhaps we just get other models first.

I have been long working on tiles. Have you tried cooperating with noise inversion tricks? I think this can be very good, with a better trained model it may be comparable to the quality of GigaGAN.

My extension is here -> https://github.com/pkuliyi2015/multidiffusion-upscaler-for-automatic1111

I will adapt your tile model to see the result and update it here.

[2blackbar]

Yes the tile model can be a saviour for upscaling and no doubles

[halr9000]

This thread is already amazing. ^ 3 amazing devs collaborating

[lllyasviel]

The inpainting model may need more considerations in implementation and perhaps we just get other models first.

I’m the author of sd-webui-segment-anything and I am planning connect my extension to your inpainting model.

So at this moment, the inpainting ControlNet cannot target at the mask only while not changing other parts, right?

my gradio demo does not have masked diffusion in it. what is displayed now is just original results from standard non-masked diffusion. but masked diffusion will be better.

[pkuliyi2015]

The model works as expected in automatic1111 txt2img; it does generate the guided content.

However, as I directly download the model and use it in this extension, it produces severe artifacts. I read the source code for a while but still not clear what should be done to make it work.

Some initial observations:

• Severe ghost shadows and duplicated contours, regardless of tile overlaps
• Faded colors in txt2img (even if with 840000 VAEs)
• Has no effect when using noise inversion (maybe this is my code flaws; I'm checking it).

See here for one result: https://imgsli.com/MTY5ODQw

[2blackbar]

what preprocessor we should use with tile controlnet model ? Using it without preprocessor gets "some" results but the resolution is kinda lower than if i would inpaint with 0.55 denoise, have to use cfg 2-3

[continue-revolution]

The inpainting model may need more considerations in implementation and perhaps we just get other models first.

I’m the author of sd-webui-segment-anything and I am planning connect my extension to your inpainting model.
So at this moment, the inpainting ControlNet cannot target at the mask only while not changing other parts, right?

my gradio demo does not have masked diffusion in it. what is displayed now is just original results from standard non-masked diffusion. but masked diffusion will be better.

Do you think there is a need to wait for an update of this extension? Is the current extension compatible with the new models, especially the inpainting model?

[lllyasviel]

The model works as expected in automatic1111 txt2img; it does generate the guided content.

However, as I directly download the model and use it in this extension, it produces severe artifacts. I read the source code for a while but still not clear what should be done to make it work.

Some initial observations:

• Severe ghost shadows and duplicated contours, regardless of tile overlaps
• Faded colors in txt2img (even if with 840000 VAEs)
• Has no effect when using noise inversion (maybe this is my code flaws; I'm checking it).

See here for one result: https://imgsli.com/MTY5ODQw

which one is cn11tile? left or right?

[pkuliyi2015]

The model works as expected in automatic1111 txt2img; it does generate the guided content.
However, as I directly download the model and use it in this extension, it produces severe artifacts. I read the source code for a while but still not clear what should be done to make it work.
Some initial observations:

• Severe ghost shadows and duplicated contours, regardless of tile overlaps
• Faded colors in txt2img (even if with 840000 VAEs)
• Has no effect when using noise inversion (maybe this is my code flaws; I'm checking it).

See here for one result: https://imgsli.com/MTY5ODQw

which one is cn11tile? left or right?

The right one. I must have done something wrong. But until now I cannot fix it.

[ProGamerGov]

Is there a PR in this repo yet for implementing ControlNet v1.1?

[lllyasviel]

The model works as expected in automatic1111 txt2img; it does generate the guided content.
However, as I directly download the model and use it in this extension, it produces severe artifacts. I read the source code for a while but still not clear what should be done to make it work.
Some initial observations:

• Severe ghost shadows and duplicated contours, regardless of tile overlaps
• Faded colors in txt2img (even if with 840000 VAEs)
• Has no effect when using noise inversion (maybe this is my code flaws; I'm checking it).

See here for one result: https://imgsli.com/MTY5ODQw

which one is cn11tile? left or right?

The right one. I must have done something wrong. But until now I cannot fix it.

from the result it looks like your input image is bigger than h/8 w/8.

for example, if you diffuse at 512 512, your tile need to be 64 64 and then use 3 cv2. pyrup to interpolate to 512.

or you can add a gaussion blur to the inputs to make it smoother