Microsoft Research has upped the game with an even smaller model. phi-1 is a transformer based model with just 1.3 billion parameters.

I wonder if higher quality datasets are the future rather than using tons of internet scraped texts. Either way, neat model!