Developing advanced AI systems like ChatGPT requires huge technical resources, in part because they are expensive to develop and operate. While several open source efforts have attempted to reverse engineer proprietary, closed-source systems created by commercial labs such as Alphabet’s DeepMind and OpenAI, they have often run into roadblocks — mainly due to a lack of capital and domain expertise.
Hoping to avoid this fate, a community research group, EleutherAIis forms a non-profit foundation. The organization announced today that it will establish a non-profit research institute, the EleutherAI Institute, funded by donations and grants from donors including AI startups Hugging Face and Stability AI, former GitHub CEO Nat Friedman, Lambda Labs and Canva.
“Formalizing as an organization allows us to build a full-time staff and participate in longer and more involved projects than would be feasible as a volunteer group,” said Stella Biderman, an AI researcher at Booz Allen Hamilton who will co-lead the EleutherAI Institute. businessupdates.org told me in an email interview. “Specifically in terms of a nonprofit, I think it’s a good idea given our focus on research and the open source space.”
EleutherAI started several years ago as a grassroots collection of developers working on open-source AI research. The founders – Connor Leahy, Leo Gao, and Sid Black – wrote the code and collected the data needed to create a machine learning model close to OpenAI’s text-generating GPT-3, which was getting a lot of attention at the time.
EleutherAI curated and open-sourced The Pile, a collection of datasets designed to be used to train GPT-3-like models to complete text, write code, and more. And it has released several models under the Apache 2.0 license, including GPT-J and GPT-NeoX, language models that fueled a whole new wave of startups for a while.
To train its models, EleutherAI relied primarily on the TPU Research Cloud, a Google Cloud program that supports projects with the expectation that the results will be shared publicly. CoreWeave, a US-based cryptocurrency miner that provides cloud services for AI workloads, also provided computing resources to EleutherAI in exchange for models for its customers to use and operate.
EleutherAI grew rapidly. Today, more than 20 of the community’s permanent staff work full-time and focus primarily on research. And in the past 18 months, EleutherAI members have co-authored 28 academic papers, trained dozens of models, and released ten codebases.
But the fickle nature of its cloud providers sometimes forced EleutherAI to scuttle its plans. Originally, the group had planned to release a model roughly the size of GPT-3 in terms of the number of parameters, but they ultimately shelved that roadmap for technical and financial reasons. (In AI, parameters are the parts of the model learned from historical training data and essentially determine the model’s proficiency for a problem, such as generating text.)
At the end of 2022, EleutherAI got acquainted with Stability AI, the now well-funded startup behind the image-generating AI system Stable Diffusion. Together with other collaborators, it helped create the first version of Stable Diffusion. And since then, Stability AI has donated some of its AWS cluster’s computing power to EleutherAI’s ongoing research into language models.
After another major patron — Hugging Face — approached EleutherAI, the nonprofit discussions began, Biderman says. (Many EleutherAI employees were involved in the company’s BigScience effort, which over the course of a year attempted to train and open source a model similar to GPT-3.)
“EleutherAI has historically focused largely on large language models that are architecturally similar to ChatGPT, and likely will continue to do so,” said Biderman. “In addition to training large language models, we are excited to dedicate more resources to ethics, interpretability, and tuning.”
One might wonder if the involvement of commercially motivated ventures such as Stability AI and Hugging Face – both of which are backed by significant venture capital – could influence EleutherAI’s research. It’s a natural assumption – and it’s even supported by evidence. At least one study shows a direct correlation between donations and the likelihood that nonprofits will speak out on a proposed government rule.
Biderman states that the EleutherAI Foundation will remain independent and says he sees no problem with the donor pool so far.
“We don’t develop models on behalf of commercial entities,” Biderman said. “In any case, I think having a diverse sponsorship improves our independence. If we were fully funded by one tech company, that seems like a much bigger potential problem on our part.”
Another challenge the EleutherAI Foundation will have to overcome is ensuring that the treasury does not run dry. OpenAI is a cautionary tale; after being incorporated as a nonprofit in 2015, the company later moved to a “capped profit” structure to fund its ongoing research.
In general, non-profit initiatives to fund AI research have been a mixed bag.
One of the success stories is The Allen Institute for AI (AI2), founded by the late Microsoft co-founder Paul Allen, which strives for scientific breakthroughs in AI and machine learning. There is also the Alan Turing Institute, the UK’s government-funded national institute for data science and machine learning. Smaller promising efforts include the AI startup’s Cohere For AI (despite its ties to the company) and Timnit Gebru’s Distributed AI Research, a globally distributed research organization.
But for every AI2, there’s former Google chairman Eric Schmidt’s fund for AI research. It was over $125 million in size and sparked new controversy after Politico reported that Schmidt wields unusually heavy influence over the White House Office of Science and Technology Policy.
Time will tell which direction the EleutherAI Foundation ultimately takes. It is likely that the mission will evolve and change over time – for the better, we can only hope.