>> You can slow things down, but not by more than a few years, because of the gradual democratization of training foundation models.
Just to be clear, what's being "democratised" is the fine-tuning of second-tier, inferior-performance models; or pre-training of third-tier ones. In the game of training large neural nets, the players that can afford to train the largest models with the most amount of data and compute at any given time will continue to dominate for the foreseeable future.
To make it plain, maybe in a couple of years you'll be able to train GPT-4 on your student laptop (unlikely, but let's allow it for the sake of argument). You'll still not be able to get anywhere near the performance of GPT-6 or whatever OpenAI and Google will be able to train by then.
Academics, hobbyists and smaller companies will continue to play second fiddle to large corporations as long as the dominant paradigm is more data and more compute.
Capabilities of the open-source models are only increasing over time by objective measurement. Yes, every one of them is demonstrably inferior to GPT-4, but we have historical precedent that the cost of compute only ever goes down.
Additionally, assuming the leaked details given here are accurate, there might not be a GPT-6. This entire approach of AI via language models very well could be approaching a local maximum and/or have already reached the point of diminishing returns.
If that is the case, OpenAI's moat is guaranteed to run dry. It should be telling that very few of the improvements over the past few months involve the base model, rather they are value-adds like plug-ins and hooking it up to a VM, things that are not protected by training difficulty.
Just to be clear, what's being "democratised" is the fine-tuning of second-tier, inferior-performance models; or pre-training of third-tier ones. In the game of training large neural nets, the players that can afford to train the largest models with the most amount of data and compute at any given time will continue to dominate for the foreseeable future.
To make it plain, maybe in a couple of years you'll be able to train GPT-4 on your student laptop (unlikely, but let's allow it for the sake of argument). You'll still not be able to get anywhere near the performance of GPT-6 or whatever OpenAI and Google will be able to train by then.
Academics, hobbyists and smaller companies will continue to play second fiddle to large corporations as long as the dominant paradigm is more data and more compute.