flux controlnet train ReadMe have a bug

[Johnson-yue]

Describe the bug

what is the controlnet config parameters? text is num_single_layers = 10, but the code set num_single_layers=0?

Reproduction

check readme file

Logs

System Info

diffusers ==0.33.0

Who can help?

No response

[hlky]

Hi @Johnson-yue, thanks for catching this. The default in the code is also different

diffusers/examples/controlnet/train_controlnet_flux.py

Lines 561 to 566 in 9352a5c

	parser.add_argument(
	"--num_single_layers",
	type=int,
	default=4,
	help="Number of single layers in the controlnet (default: 4).",
	)

cc @sayakpaul What value would be appropriate for the default? We should update it in docs, code and the example.

[sayakpaul]

Yes, I think sensible is 10. Cc: @PromeAIpro

[Johnson-yue]

I test the num_double_layers=4, and num_single_layers=10, but 80G GPU is OOM...... If your check done, you can close this issue ,thanks

[hlky]

@sayakpaul @Johnson-yue It seems this is why the example uses num_single_layers=0, let's update the docs, code and example to consistently use num_single_layers=0

[Johnson-yue]

@hlky Yes , but I mean if set num_single_layers=0, 80G GPU is still not enough , and have some other bugs ,like not successful train controlnet work . maybe should check the train code again and then update READ_flux.md and train_controlnet_flux.py

[sayakpaul]

Cc: @PromeAIpro

[yeluoo]

I use train_controlnet_flux.py, use bf16, resolution is 512, memory exceeds 80G, training loss fluctuates greatly, and the sampling during training is very unstable. What is the reason? @PromeAIpro

[Johnson-yue]

finally , someone have same problem as me

[Johnson-yue]

I do not close this issue because the doc not update

[PromeAIpro]

oom fixed, readme updated, see #11373
@yeluoo

[Johnson-yue]

@PromeAIpro I see your update， but not fix this issue problem，plz your check ReadMe_flux.md and this issue .

[PromeAIpro]

many issues have piled up together in this issue

1. fail to train a result (may because using experimental deepspeed. Refer readme and carefully modify train_controlnet_flux.py )
2. oom (fixed in fix issue that training flux controlnet was unstable and validation r… #11373. Bigger num_double_layers or num_single_layers will absolute take more vram, existing opensource controlnets say that num_single_layers not help much. So set them to proper number according to gpu memory, See readme there is a gpu usage table for referer (with num_double_layers=4, single=0).)
3. defaults are not same, in code num_single_layers=10 or 4, and in readme num_single_layers=0 (it should and would be fixed)

[Johnson-yue]

many issues have piled up together in this issue

1. fail to train a result (may because using experimental deepspeed. Refer readme and carefully modify train_controlnet_flux.py )
2. oom (fixed in fix issue that training flux controlnet was unstable and validation r… #11373. Bigger num_double_layers or num_single_layers will absolute take more vram, existing opensource controlnets say that num_single_layers not help much. So set them to proper number according to gpu memory, See readme there is a gpu usage table for referer (with num_double_layers=4, single=0).)
3. defaults are not same, in code num_single_layers=10 or 4, and in readme num_single_layers=0 (it should and would be fixed)

@PromeAIpro yes, your reply is very quick and clear, this issue talk about question 3, if default value is changed to the same as readme ,this issue will be closed

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.