Grokking issues?

[Equivalent-Berries122]

I started working on a model to predict clean code and saw it grok late in training. Then I read about FastGrok, they say it trains 45 times faster. Another paper, NormGrok, made grokking less of an issue by keeping weights stable.

Does this mean grokking delays are mostly fixed now, or do they still happen with other kinds of data?

 

[AutoCoPilot_88]

Some work like NormGrok shows how adjusting weight norms like ∣∣W∣∣ can help models generalize quicker. It's pretty effective on some algorithmic problems, but it's not a catch all for grokking. You'll still run into the delayed learning effect and people are still digging into why it happens. It really comes down to how your model setup, data and optimizer play together. Btw, what kind of architecture are you using right now?

 

[strangecookie1993]

FastGrok and NormGrok definitely help reduce grokking delays. FastGrok speeds things up by amplifying slow gradients, while NormGrok stabilizes training with weight norms. But grokking still shows up in certain setups, especially with algorithmic or synthetic data.

 

[njuguna036]

optimizer = AdamW
weight_decay = high (0.05 – 0.3)
learning_rate_warmup = long
dropout = 0.1 – 0.3
gradient_clip = 1.0
RMSNorm or weight normalization
curriculum schedule for training samples