QLoRA activation dtype

[jpmann]

In case of QLoRA, are the activations quantized to 4bit and dequantized during backward pass?

[younesbelkada]

Hi @jpmann
For QLoRA the activations are always in 16-bit precision, therefore the activations are not quantized during the forward and backward pass