Despite not being trained on a general corpus, Galactica outperforms BLOOM and OPT-175B on BIG-bench. Galactica is also significantly less toxic than other language models based on evaluations.

We train for over four epochs and experience improving performance with use of repeated tokens. For the largest 120B model, we trained for four epochs without overfitting.

Galactica performs well on reasoning, outperforming Chinchilla on mathematical MMLU by 41.3% to 35.7%, and PaLM 540B on MATH with a score of 20.4% versus 8.8%.

We release our initial paper below. We train on a large scientific corpus of papers, reference material, knowledge bases and many other sources. Includes scientific text and also scientific modalities such as proteins, compounds and more. galactica.org/paper.pdf

🪐 Introducing Galactica. A large language model for science. Can summarize academic literature, solve math problems, generate Wiki articles, write scientific code, annotate molecules and proteins, and more. Explore and get weights: galactica.org

SoTA in the LRA benchmark is achieved by Mega (Ma et al. 2022), a single-head gated attention mechanism equipped with exponential moving average to incorporate inductive bias of position-aware local dependencies into the position-agnostic attention mechanism. 8/10

There’s also interest in assessing speed and memory footprint of ML models. Tay et al. (2020) reported efficiency results for a set of models. Performer and Linear Transformer make a better trade-off in terms of speed and performance with reasonable memory usage. 7/10

There exists many solutions that have been built and designed to perform well at long-range modeling. Some notable transformer-based models include BigBird, Performer, Longformer, Synthesizer, among others. Check performance of these models here: paperswithcode.com/sota/long… 5/10

The LRA benchmark includes examples of sequences that range from 1K to 16K tokens. It consists of different data types and modalities such as text, synthetic images, mathematical expressions and so on. (Figure below shows ListOPs task included in the LRA benchmark) 4/10

One of the most recent challenging benchmarks to measure capabilities of language models is called BIG-bench. It already includes 204 tasks! It includes tasks like detecting misconceptions, dark humor detection, high school physics/biology, and more. paperswithcode.com/dataset/b… 2/6