Contributing to the Image Generation Benchmark

[BenjaminBossan]

A place to discuss community contributions to the image generation benchmark. Before starting your experiments, read the contribution guideline and discuss your ideas with the maintainers here.

The results can be seen in this Gradio Space (on the top left, select the image-gen task). Failed experiments (i.e. ones that didn't lead to an improvement) go to the benchmark graveyard -- check it to see past experiments and upload your failed experiments there.

[Sergasgr]

Hi @BenjaminBossan,

I'd like to propose adding a BEFT experiment to the Image Generation Benchmark. I already ran it locally, so I'm sharing the full picture — reasoning and results, including a comparison against the existing LoRA/OFT runs — in one message.

Why BEFT
It wasn't covered in experiments/ yet. Since the DreamBooth "cat pillow" dataset is a few-shot scenario, BEFT feels like a natural gap to fill. The paper specifically reports strong results in low-data regimes, and it also showed a solid memory/accuracy trade-off in the MetaMathQA benchmark recently.

(Side note: This work used AI assistance for configuration review and result analysis. I personally reviewed it, ran the experiment myself, and everything below comes from my own run.)

Setup

• Model: black-forest-labs/FLUX.2-klein-base-4B
• target_modules: matched exactly to the existing LoRA/OFT configs for a fair comparison — to_q, to_k, to_v, to_out.0, add_q_proj, add_k_proj, add_v_proj, to_add_out, to_qkv_mlp_proj, linear_in, linear_out
• Hardware: RTX 5070 Ti, 16GB (Note: existing LoRA/OFT results were run on an L40S, so runtime below isn't a direct head-to-head)

Minor bug report: While copying the target_modules, I noticed "to_add_out" is listed twice in lora/flux2-klein-default/adapter_config.json. I know duplicate entries are a no-op, but I wanted to flag it for a potential quick cleanup. I removed the duplicate for my BEFT run.

Results & Comparison
Here is how BEFT compares against the existing LoRA (r=32) and OFT runs:

Metric	LoRA (r=32)	OFT	BEFT
DINOv2 similarity (test)	0.6972	0.7078	0.5227
Drift	0.2620	0.2619	0.0732
Max memory reserved	10212 MB	9222 MB	9458 MB
Adapter checkpoint size	146.3 MB	18.2 MB	3.4 MB
Trainable params	38,338,560 (0.98%)	4,761,600 (0.12%)	890,880 (0.023%)
Runtime (train)	759.2s (L40S)	524.0s (L40S)	989.8s (RTX 5070 Ti)

An honest read on the trade-offs:
BEFT trades some subject fidelity for a massive drop in trainable parameters (43x smaller than LoRA, 5x smaller than OFT). Interestingly, it shows notably lower drift than both (0.073 vs ~0.262), meaning the base "cat" concept stays much more intact. Memory doesn't scale down proportionally, but that's expected since the frozen 4B base model's activations dominate the VRAM.

Qualitatively, the subject was still very recognizable (I'm attaching a photo and a color drawing sample). I also noticed that validation similarity climbed to 0.530 around step 400 and oscillated in the 0.50–0.52 range through step 700. Since it seemed to plateau early, pushing max_steps further didn't feel necessary for this first pass.

Next Steps
Since this is my first time contributing to this benchmark, I'd love to get your perspective on these numbers. Does my reading of the trade-offs make sense to you? Let me know if you think this profile makes BEFT a worthwhile addition to the current experiments.

[BenjaminBossan]

Thanks for the submissions to the graveyard, I replied to those. Regarding UniLoRA and those LoRA variants: Both are fair game, feel free take a stab. Regarding HRA, I would currently not experiment with it because the implementation is rather inefficient. We're thinking about refactoring it.

[Sergasgr]

Hi @BenjaminBossan,

I ran a small ablation sweep for UniLoRA since the paper doesn't cover image generation at all. Fixed r=4 (the library default) and swept theta_d_length:

Config	DINOv2	Drift	Trainable params
theta_d=100k	0.3912	0.0383	100,000
theta_d=250k	0.4394	0.0572	250,000
theta_d=500k	0.6562	0.2055	500,000
theta_d=750k	0.6597	0.2359	750,000

This reproduces the paper's own diminishing-returns ablation almost exactly: a steep climb up to 500k (+0.265 DINOv2), then a plateau (+0.0035 from 500k to 750k) — and past that plateau, drift keeps rising (0.206 to 0.236) with no fidelity gain, so 750k has a real cost with no benefit over 500k.

I also tested whether bumping r helps the way it did for HiRA. At the exact same theta_d_length=500k (so the same 500,000 trainable params either way), r=32 scores 0.4462 vs r=4's 0.6562 — 32% worse — and uses 410MB more memory despite identical trainable params. My read: unlike LoRA's r, UniLoRA's r doesn't add capacity when theta_d_length is fixed — it just makes each layer draw a bigger slice from the same fixed shared vector, so more layers end up reusing (aliasing) the same entries instead of having distinct information to work with. So r=4 (the default) is the right call here, for a real mechanistic reason rather than just "the library default happened to work."

Proposing r=4, theta_d_length=500,000 as the entry, and the other four runs (100k, 250k, 750k, and the r=32 comparison) for the benchmark graveyard. I'll open the PRs once you confirm this reading looks right.

Separately: planning to run VeLoRA next (as a LoRA velora_config variant, testing memory reduction vs the existing LoRA r=32 baseline). Nothing to report yet, will follow up once I have results.

[BenjaminBossan]

Thanks for running the UniLoRA experiments, your proposal sounds good, please proceed.

[Sergasgr]

Hi @BenjaminBossan,

Opened the UniLoRA PR (#3740) with the r=4, theta_d_length=500,000 config from the ablation discussed above, and also opened the corresponding PR to the benchmark-graveyard.

Also ran three VeLoRA experiments (as a velora_config on top of LoRA):

Config	DINOv2	Drift	Memory	Trainable params
r=32, 11 modules (standard)	0.6907	0.2588	10332 MB	38,338,560
r=32, Value+Down only	0.6568	0.2375	9396 MB	5,898,240
r=64, 11 modules	0.6886	0.2554	11050 MB	76,677,120

Honest read: the memory savings are much smaller than the paper reports (34MB vs the existing LoRA r=32 baseline, compared to the paper's 7-18% reductions). My best guess is that this benchmark already runs with gradient checkpointing enabled by default, which addresses the same activation-memory problem VeLoRA targets, just via recomputation instead of compression — the paper's own benchmarks don't use gradient checkpointing, so this specific interaction was never tested there. The r=64 run reinforces this: it doubles trainable params exactly but uses 718MB more memory with no quality gain, since VeLoRA's savings are fixed while LoRA's own memory cost scales with r — the opposite of the paper's "afford a higher rank" argument, likely because at this model's scale (4B params) LoRA's own footprint from a rank increase outweighs what a fixed activation compression can offset.

The Value+Down-only run isn't a clean isolation of VeLoRA's effect either, since it also narrows LoRA's own target_modules from 11 to 3 — some of its memory reduction likely comes from adapting fewer layers, not from the compression itself.

Proposing the standard r=32/11-module run as the entry (best quality, cleanest comparison to the existing LoRA baseline), and the other two for the graveyard.

[Sergasgr]

Hi @BenjaminBossan,

I've opened two PRs for the LoRA variants: BD-LoRA (#3799) and MonteCLoRA (#3806). The superseded variants from each (3 for BD-LoRA, 2 for MonteCLoRA) are also up in the benchmark-graveyard dataset, referenced from their respective PRs.

I'm open to updating the HRA experiment in image-gen and adding a new one for MetaMathQA once the refactor is done.

[Mr-J-Forest]

Hi @BenjaminBossan, I would like to contribute one additional PVeRA configuration to the image-generation benchmark.

Proposed scope

• Method: PVeRA, which currently has only the default r=32, d_initial=0.1 image-generation experiment.
• Change exactly one adapter hyperparameter: test d_initial=0.05; keep the base model, target modules, rank, optimizer, 750-step budget, and all benchmark settings unchanged.
• Rationale: d_initial controls the initial scaling vector in PVeRA. A smaller value may make the early update magnitude less aggressive while preserving PVeRA's parameter count, checkpoint format, and memory profile.
• Evaluation: reproduce the baseline as far as local hardware permits, then run the candidate on suitable GPU hardware. I will report test DINO similarity, drift, train time, peak memory, and failed results to the benchmark graveyard rather than opening a PR if it is not useful.

My local GPU has 8 GB VRAM while the existing benchmark peaks at about 10.6 GB, so I will not treat a reduced-memory exploratory run as comparable to the official result. I will only submit the configuration for the maintainers' standardized rerun.

AI assistance is being used for repository research and experiment planning. I will personally review all changes, run the experiments, and verify the final report. Does this narrow scope look useful before I start?

[BenjaminBossan]

Thanks for offering your help.

My local GPU has 8 GB VRAM while the existing benchmark peaks at about 10.6 GB, so I will not treat a reduced-memory exploratory run as comparable to the official result. I will only submit the configuration for the maintainers' standardized rerun.

IIUC, you would not run the experiment at all? In that case, the contribution wouldn't really add value, how would you know what works better and what doesn't? AFAIK, you could use free Kaggle notebooks with 16 GB GPUs to run the experiments, perhaps that's an option for you.