adaboundw

[Laksh1997]

Hi - I thought I would raise this PR which implements AdaBoundW.

I did so because I was using AdaBound from this repo with weight_decay=0.1 and noticed the training loss couldn't go down. The training also failed with Adam but succeeded with AdamW.

The algorithm is very simple, in fact just 1 line added compared to the original version:

        for group, base_lr in zip(self.param_groups, self.base_lrs):
            for p in group['params']:
                if p.grad is None:
                    continue

                # Perform stepweight decay
                p.mul_(1 - base_lr * group['weight_decay'])

Note, in the original implementation, the weight_decay is not scaled by the base_lr (which I think is not common practice as the official PyTorch Repo does it this way with AdamW (https://pytorch.org/docs/stable/_modules/torch/optim/adamw.html#AdamW)

[jettify]

Should this be named AdaBoundW?

[Laksh1997]

Yes, of course - my bad!

[Laksh1997]

Have changed to AdaBoundW now