Style Transfer Sometimes Confuses Content and Style Images

[fikry102]

Issue Description

When using this repo for style transfer, I occasionally observe that the model confuses
the content image and the style image — the output appears to transfer the content of
Figure 2 onto Figure 1's style, rather than the intended direction.

Prompt Used

Style Transfer the style of Figure 2 to Figure 1, and keep the content and
characteristics of Figure 1.

Expected Behavior

• Style source: Figure 2
• Content source: Figure 1
• Output should preserve the content, layout, and structure of Figure 1,
  while adopting the visual style of Figure 2.

Observed Behavior

In some runs, the output seems to swap the roles of the two figures —
the content of Figure 2 appears in the output instead of Figure 1's content.

Questions

1. Is there a known reason why the model occasionally swaps content and style references?
2. Is the confusion related to prompt sensitivity or image ordering?

Looking forward to your feedback, thanks!

[ShiwenZhang-teleai]

Hello, I just tried your example, here is my result with the public demo I provided, it seems working. I tried min edge=1024 and 832 and they work fine. Maybe you could try a different min edge and see whether it works.

However, I did observe the problem you mentioned sometimes with other cases I tried myself. This is a problem that we will try to solve in the future by post training to stablize the model performance.

[fikry102]

Thanks for your reply. In my case, the min_edge is set to 512.

All content images are 512 × 512. The style images vary in aspect ratio. (let me give the content image and style image of the above case)

I noticed that the style image is resized to min_edge × min_edge. I tried changing this logic to preserve the original aspect ratio as much as possible while ensuring that both height and width are multiples of 16, and found that this fixes some cases. However, some cases are still not resolved. (One is given below)

Let me provide the corresponding content and style image for the convenience of testing.