Skip empty MultiLoraAdapter when no LoRAs target a model

[fszontagh]

Summary

apply_loras_at_runtime() always wraps each model (cond_stage_model, diffusion_model, first_stage_model) with a MultiLoraAdapter, even when no LoRA tensors actually match that model's prefix. With an empty adapter the model still routes every Linear/Conv2d::forward() through WeightAdapter::forward_with_lora() rather than the direct ggml_ext_linear / ggml_ext_conv_2d path:

// ggml_extend.hpp Linear::forward
if (ctx->weight_adapter) {
    // -> MultiLoraAdapter::forward_with_lora -> empty patch_weight loop -> ggml_ext_linear
    return ctx->weight_adapter->forward_with_lora(...);
}
return ggml_ext_linear(...);    // direct path

For a typical character LoRA targeting only the diffusion model, this means the cond_stage (CLIP/T5/LLM) and first_stage (VAE) graphs both go through the indirect path for no benefit — every patch_weight call iterates an empty lora_models vector and the inner ggml_ext_linear/ggml_ext_conv_2d is invoked from inside the adapter rather than directly.

The fix is to only attach the adapter when its lora_models list is non-empty. set_weight_adapter(nullptr) is already called at the top of apply_loras_at_runtime, so skipping the later assignment leaves the adapter correctly cleared.

Test plan

• Builds cleanly off master
• Image generation still works for a Z-Image LoRA targeting only the diffusion model — image is bit-identical to before (the diffusion adapter is non-empty, so its path is unchanged)
• When LoRA targets only diffusion, cond_stage_model and first_stage_model no longer carry a stale MultiLoraAdapter
• Reviewer to confirm desired behavior when subsequent calls add cond_stage/first_stage LoRAs (the adapter will be (re)attached on the next apply_loras_at_runtime call, which I believe is the intended flow)