Dimitris Tsipras retweeted
We have reached an agreement in principle for Sam Altman to return to OpenAI as CEO with a new initial board of Bret Taylor (Chair), Larry Summers, and Adam D'Angelo.
We are collaborating to figure out the details. Thank you so much for your patience through this.
Dimitris Tsipras retweeted
We know that language models (LMs) reflect opinions - from internet pre-training, to developers and crowdworkers, and even user feedback. But whose opinions actually appear in the outputs? We make LMs answer public opinion polls to find out: arxiv.org/abs/2303.17548
Dimitris Tsipras retweeted
@aleks_madry testifying at the Congressional subcommittee on AI technology! That's my postdoc advisor!!
oversight.house.gov/hearing/…
Dimitris Tsipras retweeted
Data selection for LMs (GPT-3, PaLM) is done with heuristics that select data by training a classifier for high-quality text. Can we do better?
Turns out we can boost downstream GLUE acc by 2+% by adapting the classic importance resampling algorithm..
arxiv.org/abs/2302.03169
🧵
Dimitris Tsipras retweeted
For this year's CS 224n: Natural Language Processing with Deep Learning, I've written notes on our Self-Attention and Transformers lecture.
web.stanford.edu/class/cs224…
Topics: Problems with RNNs, then self-attention, then a 'minimal' self-attention architecture, then Transformers.
Dimitris Tsipras retweeted
Can transformers follow instructions? We explore this in:
"Looped Transformers as Programmable Computers"
arxiv.org/abs/2301.13196
led by Angeliki (@AngelikiGiannou) and Shashank (@shashank_r12) in collaboartion with the @Kangwook_Lee and @jasondeanlee
Here is a 🧵
Dimitris Tsipras retweeted
Recent events (ahem) have brought the debate on whether/how to regulate social media back to the forefront.
My students @cen_sarah @andrew_ilyas and I have been thinking about this for a *while*. Excited to share the first results of our thinking: aipolicy.substack.com/p/soci… (1/3)
Dimitris Tsipras retweeted
📣 CRFM announces PubMedGPT, a new 2.7B language model that achieves a new SOTA on the US medical licensing exam. The recipe is simple: a standard Transformer trained from scratch on PubMed (from The Pile) using @mosaicml on the MosaicML Cloud, then fine-tuned for the QA task.
Dimitris Tsipras retweeted
Stable diffusion can visualize + improve model failure modes!
Leveraging our method, we can generate examples of hard subpopulations, which can then be used for targeted data augmentation to improve reliability.
Blog: gradientscience.org/failure-…
@saachi_jain_ @HLawrenceCS A.Moitra
Our #NeurIPS2022 poster on in-context learning will be tomorrow (Thursday) at 4pm! Come talk to @shivamg_13 and me at poster #928 🔥
LLMs can do in-context learning, but are they "learning" new tasks or just retrieving ones seen during training? w/ @shivamg_13, @percyliang, & Greg Valiant we study a simpler Q:
Can we train Transformers to learn simple function classes in-context? 🧵
arxiv.org/abs/2208.01066
Dimitris Tsipras retweeted
You’re deploying an ML system, choosing between two models trained w/ diff algs. Same training data, same acc... how do you differentiate their behavior?
ModelDiff (gradientscience.org/modeldif…) lets you compare *any* two learning algs!
w/ @harshays_ @smsampark @andrew_ilyas (1/8)
Dimitris Tsipras retweeted
Language models are becoming the foundation of language technologies, but when do they work or don’t work? In a new CRFM paper, we propose Holistic Evaluation of Language Models (HELM), a framework to increase the transparency of LMs. Holistic evaluation includes three elements:
Dimitris Tsipras retweeted
arxiv.org/abs/2210.15097
We propose contrastive decoding (CD), a more reliable search objective for text generation by contrasting LMs of different sizes. CD takes a large LM (expert LM e.g. OPT-13b) and a small LM (amateur LM e.g. OPT-125m) and maximizes their logprob difference
Dimitris Tsipras retweeted
How might an online community look after many people join? My paper w/ @lindsaypopowski @Carryveggies @merrierm @percyliang @msbernst introduces "social simulacra": a method of generating compelling social behaviors to prototype social designs 🧵
arxiv.org/abs/2208.04024 #uist2022
LLMs can do in-context learning, but are they "learning" new tasks or just retrieving ones seen during training? w/ @shivamg_13, @percyliang, & Greg Valiant we study a simpler Q:
Can we train Transformers to learn simple function classes in-context? 🧵
arxiv.org/abs/2208.01066
Overall, it’s clear that Transformers can be trained to encode far-from-trivial learning algorithms, but *how* they encode such algorithms is still a wide open question.
See our paper for additional experiments and many more open questions [arxiv.org/abs/2208.01066]!