I forgot to change the default value of discrete_flow_shift to 1.0 in the anima_train_network.md document. Would you mind updating it? Thanks!

I forgot to change the default value of discrete_flow_shift to 1.0 in the anima_train_network.md document. Would you mind updating it? Thanks!

Of course, thank you for letting me know!

I'd like to replace Anima's VAE code with Qwen Image VAE, licensed under ASL 2.0 from Diffusers.

I would like to change the LoRA module selection to be regular expression based, as I think it would be more flexible.

I don't see the need to use fp32 for transformer weights for LoRA training, and there are other options for fp8. Also, finetuning defaults to fp32 and can optionally change to bf16/fp16, so I'll remove the transformer_dtype option.

I will change it to use existing options such as --pretrained_model_name_or_path.

Also, it seems that caption dropout is effective in Anima. However, I think it will be confusing if caption_dropout_rate of the dataset is ignored, so I will consider some way to address this.

I'd like to use sd3_train_utils.get_noisy_model_input_and_timesteps instead of anima_train_utils.get_noisy_model_input_and_timesteps to support min/max timesteps etc. Please let me know if there are any issues with replacing it.

Also, replacing process_batch in AnimaNetworkTraineer involves a lot of overlapping functionality, so I plan to do something about it.

I don’t see any issues with making that replacement. Some of the functions in anima_train_utils were copied or adapted from sd3_train_utils in the first place, so using sd3_train_utils.get_noisy_model_input_and_timesteps should be fine as long as it fits your training setup.

The overlapping functionality was intentional to help avoid conflicts with other training models. If you plan to refactor process_batch to reduce duplication, that sounds like a good improvement.

If you have any question about my implement. Feel free to ask i am happy to help you.

The following processing is performed using LLMAdapter.

  crossattn_emb = self.llm_adapter(
      source_hidden_states=crossattn_emb,
      target_input_ids=t5_input_ids,
      target_attention_mask=t5_attn_mask,
      source_attention_mask=source_attention_mask,
  )
  if t5_attn_mask is not None:
      crossattn_emb[~t5_attn_mask.bool()] = 0

When the uncond string "" is tokenized, all of Qwen3's attention masks become 0. So, crossattn_emb becomes all 0.

Therefore, when uncond, no gradients to the LLMAdapter. It also seems that we do not need to cache the uncond embedding because it's all zero always.

Please let me know if there is anything in my understanding that is incorrect.

Your understanding is incorrect. Here's why:

Qwen3 tokenizing "" does NOT produce all-zero attention mask.

Qwen3 has eos_token = "<|endoftext|>" (ID 151643), no BOS. So tokenizing "" with padding="max_length", max_length=512 gives:

• tokens: [151643, 0, 0, ..., 0] (1 EOS + 511 pad)
• attn_mask: [1, 0, 0, ..., 0] — one position is 1

Same for T5: eos_token_id=1, so mask is also [1, 0, ..., 0].

After LLM Adapter + zero-out:

crossattn_emb[~t5_attn_mask.bool()] = 0

Position 0 (EOS) keeps its value from the adapter. Only positions 1-511 become zero.

Result: uncond crossattn_emb = [non_zero_EOS_embedding, 0, 0, ..., 0], not all zeros. The LLM Adapter still receives gradients through position 0, and the uncond embedding must be properly cached.

Thank you!

My simple test returns Text: [''], Qwen3 mask sum: 0, T5 mask sum: 1.

        # Tokenize with Qwen3
        qwen3_encoding = self.qwen3_tokenizer.batch_encode_plus(
            text, return_tensors="pt", truncation=True, padding="max_length", max_length=self.qwen3_max_length
        )
        qwen3_input_ids = qwen3_encoding["input_ids"]
        qwen3_attn_mask = qwen3_encoding["attention_mask"]

        # Tokenize with T5 (for LLM Adapter target tokens)
        t5_encoding = self.t5_tokenizer.batch_encode_plus(
            text, return_tensors="pt", truncation=True, padding="max_length", max_length=self.t5_max_length
        )
        t5_input_ids = t5_encoding["input_ids"]
        t5_attn_mask = t5_encoding["attention_mask"]

        print(f"Text: {text}, Qwen3 mask sum: {qwen3_attn_mask.sum().item()}, T5 mask sum: {t5_attn_mask.sum().item()}")

T5 has a mask of length 1. It's my misunderstanding.

However, all outputs of Qwen3 are masked and are therefore 0. So should we only cache T5's input_ids (1, 0, 0, ...)?

Oh, I misunderstood Qwen3 as well, I was just reading the config and making assumptions based on my knowledge. I tested it, and you're right. Your idea makes sense; I think we should cache only T5's input_ids.

@duongve13112002

When caption dropout is enabled, llm_adapter seems to return NaN for empty string regardless of whether it is cached or not. This also happens during inference.

Do you have any idea? I'm really confused...

When caption dropout is enabled, llm_adapter seems to return NaN for empty string regardless of whether it is cached or not. This also happens during inference.

Do you have any idea? I'm really confused...

This seems to be a mathematical stability issue with scaled_dot_product_attention when mask is all 0. I will change it to skip SDPA when mask is 0.

@kohya-ss Hi, i wonder when you finish refactor the code i am looking forward to use this

It's almost done so I think I'll merge it tomorrow.

Everything looks good so far. I reviewed all your changes and tested them, and I didn’t find any potential bugs.

Everything looks good so far. I reviewed all your changes and tested them, and I didn’t find any potential bugs.

I'm glad to hear that! Thank you again for your great contribution!