CS @ Stanford. Building @mlfoundry. Prev @AWS, @Lux_Capital, @UniofOxford
Joined July 2020
- Tweets988
- Following2.7K
- Followers2.7K
- Likes9.7K
Pinned Tweet
Introducing ambientGPT: an open-source and multimodal MacOS foundation model GUI
Run GPT-4o and open-source models with full ambient knowledge of your screen.
Foundation models have long been confined to the browser. With ambientGPT, your screen context is directly inferred as part of the query, ensuring you never need to explicitly upload context again!
Github: github.com/siddrrsh/ambientG…
Re Llama3V: Firstly, we want to apologize to the original authors of MiniCPM.
@AkshGarg03 and I posted Llama3V with @mustafaaljadery. Mustafa wrote the code for the project. Aksh and I were both excited about multimodal models and liked the architectural extensions on top of Idefics and Siglip that he described to us. Our role here was to help him promote the model on medium and twitter. We looked at recent papers to validate the novelty of the work but we were not informed of or made aware of any of the previous work by @OpenBMB. After seeing the twitter posts about this topic yesterday, we asked Mustafa about proof of originality for Llama3V and asked for the training code but we haven’t seen any response so far. We were waiting for Mustafa to take the lead but instead we are releasing our own statement.
We apologize to the authors of miniCBM for any inconvenience that we caused for not doing the full diligence to verify and peer review the novelty of this work. Going forward, we will be cautious and diligent, and we sincerely thank the community for bringing this to our attention. We've taken all references to Llama3V down and we apologize once again for the inconvenience we may have caused.
- Siddharth and Aksh
Shocked! Llama3-V project from a Stanford team plagiarized a lot from MiniCPM-Llama3-V 2.5!
its code is a reformatting of MiniCPM-Llama3-V 2.5, and the model's behavior is highly similar to a noised version of MiniCPM-Llama3-V 2.5 checkpoint.
Evidence: github.com/OpenBMB/MiniCPM-V…
Siddharth Sharma retweeted
Sakana AI is proud to sponsor the LLM Merging Competition: Building LLMs Efficiently through Merging at #NeurIPS2024 🤗
If you’re excited about pushing the frontiers of model merging, please visit: llm-merging.github.io/
ALT Aims and Focus Training high-performing large language models (LLMs) from scratch is a notoriously expensive and difficult task, costing hundreds of millions of dollars in compute alone. These pretrained LLMs, however, can cheaply and easily be adapted to new tasks via fine-tuning, leading to a proliferation of models that suit specific use cases. Recent work has shown that specialized fine-tuned models can be rapidly merged to combine capabilities and generalize to new skills. Get started The competition will provide the participants with a list of expert models that have already been trained on a task-specific dataset. All of these models will be publicly available on the Hugging Face Model Hub with licenses that permit their use for research purposes. These models can either be fully fine-tuned models or models obtained by parameter-efficient fine-tuning methods such as LoRA.
Will be an awesome event!
Updates w.r.t. the upcoming Compound AI Systems Workshop (June 13th in San Francisco):
Accepted Posters
We are excited to announce the accepted posters for the Compound AI Systems workshop. Due to space constraints, we were only able to accept 28 featured posters. Sincere congratulations to the authors of the accepted works 🎉. Thank you to all who submitted -- we look forward to hopefully featuring many of your works in future events.
Please see the site for the full list of accepted papers: sites.google.com/view/compou…!
Speakers
We are excited to announce the full slate of speakers:
@RichardSocher . Founder and CEO of You.com and NLP pioneer.
@MonicaSLam. Professor of CS at Stanford and director of OVAL lab.
@polynoamial . Research Scientist at OpenAI and pioneer in multi-step reasoning, self-play, and multi-agent AI.
@lmthang. Researcher at DeepMind, and PI for AlphaGeometry.
@ysu_nlp. Faculty at The Ohio State University and PI for many exciting recent works on web agents and beyond.
@hwchase17. creator of @langchain, a leading OSS developer framework for building context-aware reasoning apps.
@thismattbell. Head of Applied Research at Anthropic, building systems to support agentic workflows.
@YejinChoinka. Professor of CS at UW, leading PI at the intersection of NLP and reasoning.
@HannaHajishirzi. Professor of CS at UW, focus on multi-hop reasoning, symbolic methods, and much more.
@maithra_raghu. Founder and CEO of Samaya AI, an AI-powered knowledge-discovery platform.
The Industry + Academia panel will be moderated by @matei_zaharia, who sits uniquely at that intersection as CTO at @Databricks and CS prof at @UCBerkeley.
Registration
Also, general registration is now open. There are only ~200 seats so register at your earliest convenience. Registration is free (see instructions on the website).
Note that we have reserved spots for up to 2 authors for each accepted poster, so authors can forgo general registration.
When he's in the zone, Carlos Alcaraz is the best player in the world. The defense-to-offense and style of attacking tennis is just brutal.
ML research is the most "empirical" science to ever exist. we test things and when they work or look promising, we double click ... so a lot of major results are child nodes of a tree of trial and error
We’d like to thank the folks at Meta for their work in ensuring open-source is here to stay. We also wanted to shout out the authors of LLaVA-UHD as our methods are directly inspired by their intuition when it comes to image splitting and prepending the latents to the text. @astonzhangAZ @joespeez @ylecun @imhaotian @lukede0
Siddharth Sharma retweeted
I've spent the past ~3 weeks going through the entire history of deep learning and reimplementing all the core breakthroughs.
It has completely changed my beliefs about deep learning progress and where we're headed.
Progress tracker in thread (all resources at the end) 👇
Siddharth Sharma retweeted
This week, we showed how altering internal "features" in our AI, Claude, could change its behavior.
We found a feature that can make Claude focus intensely on the Golden Gate Bridge.
Now, for a limited time, you can chat with Golden Gate Claude: claude.ai
Great to see a community developing around this! @ollama, @GroqInc, and @vllm_project integrations on the way 🫡
Introducing ambientGPT: an open-source and multimodal MacOS foundation model GUI
Run GPT-4o and open-source models with full ambient knowledge of your screen.
Foundation models have long been confined to the browser. With ambientGPT, your screen context is directly inferred as part of the query, ensuring you never need to explicitly upload context again!
Github: github.com/siddrrsh/ambientG…
ambientGPT is open-source and we plan to integrate vllm and ollama to provide more extensive inference hosting abilities with our multimodal GUI. We also aim to release ambientGPT on the apple app store soon.
I love GPU pods
1/ @SohamGovande, @jameszhou02, @jerryzhou and I spent the weekend building PodPlex: A platform for distributed training & serverless inference at scale
I'm very glad to say that we left $10,000 GPU credits richer and 36 hours of sleep poorer
more details in 🧵
Excellent writing on decentralized/distributed training. Exciting times ahead!
1/ As promised, here's my thesis on the future of decentralized training of foundation models. Covers:
1) why decentralized makes sense from scaling, margins, and marketplace lenses
2) challenges
3) exciting enabling research shifts
In long form at: aksh-garg.medium.com/shard-o…
More thoughts in 🧵
