For others, it is a matter of profit. Stability AI hopes to repeat the same trick with chatbots that it pulled with images: fuel and then benefit from a burst of innovation among developers that use its products. The company plans to take the best of that innovation and roll it back into custom-built products for a wide range of clients. “We stoke the innovation, and then we pick and choose,” says Emad Mostaque, CEO of Stability AI. “It’s the best business model in the world.”

Either way, the bumper crop of free and open large language models puts this technology into the hands of millions of people around the world, inspiring many to create new tools and explore how they work. “There’s a lot more access to this technology than there really ever has been before,” says Biderman.

“The incredible number of ways people have been using this technology is frankly mind-blowing,” says Amir Ghavi, a lawyer at the firm Fried Frank who represents a number of generative AI companies, including Stability AI. “I think that’s a testament to human creativity, which is the whole point of open-source.”

Melting GPUs

But training large language models from scratch—rather than building on or modifying them—is hard. “It’s still beyond the reach of the vast majority of people,” says Mostaque. “We melted a bunch of GPUs building StableLM.”

Stability AI’s first release, the text-to-image model Stable Diffusion, worked as well as—if not better than—closed equivalents such as Google’s Imagen and OpenAI’s DALL-E. Not only was it free to use, but it also ran on a good home computer. Stable Diffusion did more than any other model to spark the explosion of open-source development around image-making AI last year.

This time, though, Mostaque wants to manage expectations:  StableLM does not come close to matching GPT-4. “There’s still a lot of work that needs to be done,” he says. “It’s not like Stable Diffusion, where immediately you have something that’s super usable. Language models are harder to train.”

Another issue is that models are harder to train the bigger they get. That’s not just down to the cost of computing power. The training process breaks down more often with bigger models and needs to be restarted, making those models even more expensive to build.

In practice there is an upper limit to the number of parameters that most groups can afford to train, says Biderman. This is because large models must be trained across multiple different GPUs, and wiring all that hardware together is complicated. “Successfully training models at that scale is a very new field of high-performance computing research,” she says.

The exact number changes as the tech advances, but right now Biderman puts that ceiling roughly in the range of 6 to 10 billion parameters. (In comparison, GPT-3 has 175 billion parameters; LLaMA has 65 billion.) It’s not an exact correlation, but in general, larger models tend to perform much better.

Biderman expects the flurry of activity around open-source large language models to continue. But it will be centered on extending or adapting a few existing pretrained models rather than pushing the fundamental technology forward. “There’s only a handful of organizations that have pretrained these models, and I anticipate it staying that way for the near future,” she says.

That’s why many open-source models are built on top of LLaMA, which was trained from scratch by Meta AI, or releases from EleutherAI, a nonprofit that is unique in its contribution to open-source technology. Biderman says she knows of only one other group like it—and that’s in China.

EleutherAI got its start thanks to OpenAI. Rewind to 2020 and the San Francisco–based firm had just put out a hot new model. “GPT-3 was a big change for a lot of people in how they thought about large-scale AI,” says Biderman. “It’s often credited as an intellectual paradigm shift in terms of what people expect of these models.”

Excited by the potential of this new technology, Biderman and a handful of other researchers wanted to play with the model to get a better understanding of how it worked. They decided to replicate it.

OpenAI had not released GPT-3, but it did share enough information about how it was built for Biderman and her colleagues to figure it out. Nobody outside of OpenAI had ever trained a model like it before, but it was the middle of the pandemic, and the team had little else to do. “I was doing my job and playing board games with my wife when I got involved,” says Biderman. “So it was relatively easy to dedicate 10 or 20 hours a week to it.”

Their first step was to put together a massive new data set, containing billions of passages of text, to rival what OpenAI had used to train GPT-3. EleutherAI called its dataset the Pile and released it for free at the end of 2020.

EleutherAI then used this data set to train its first open-source model. The largest model EleutherAI trained took three and a half months and was sponsored by a cloud computing company. “If we’d paid for it out of pocket, it would have cost us about $400,000,” she says. “That’s a lot to ask for a university research group.”

Helping hand

Because of these costs, it’s far easier to build on top of existing models. Meta AI’s LLaMA has fast become the go-to starting point for many new open-source projects. Meta AI has leaned into open-source development since it was set up by Yann LeCun a decade ago. That mindset is part of the culture, says Pineau: “It’s very much a free-market, ‘move fast, build things’ kind of approach.”

Pineau is clear on the benefits. “It really diversifies the number of people who can contribute to developing the technology,” she says. “That means that not just researchers or entrepreneurs but civil governments and so on can have visibility into these models.”

Like the wider open-source community, Pineau and her colleagues believe that transparency should be the norm. “One thing I push my researchers to do is start a project thinking that you want to open-source,” she says. “Because when you do that, it sets a much higher bar in terms of what data you use and how you build the model.”

But there are serious risks, too. Large language models spew misinformation, prejudice, and hate speech. They can be used to mass-produce propaganda or power malware factories. “You have to make a trade-off between transparency and safety,” says Pineau.

For Meta AI, that trade-off might mean some models do not get released at all. For example, if Pineau’s team has trained a model on Facebook user data, then it will stay in house, because the risk of private information leaking out is too great. Otherwise, the team might release the model with a click-through license that specifies it must be used only for research purposes.

This is the approach it took for LLaMA. But within days of its release, someone posted the full model and instructions for running it on the internet forum 4chan. “I still think it was the right trade-off for this particular model,” says Pineau. “But I’m disappointed that people will do this, because it makes it harder to do these releases.”