Arvind Sundararajan (@arvinds) header
Arvind Sundararajan (@arvinds) profile picture

CTO & Co-Founder @abacusai

San Francisco, CA
Joined April 2007
Photo by Arvind Sundararajan (@arvinds)Photo by Arvind Sundararajan (@arvinds)Photo by Arvind Sundararajan (@arvinds)Photo by Arvind Sundararajan (@arvinds)Photo by Arvind Sundararajan (@arvinds)
Arvind Sundararajan retweeted
Pretty much 😂
I read the DeepSeek papers and I think. “None of this is complicated. I’m just way too lazy to write my own Cuda code. I’m American. This is America. Programming your GPU at 100% efficiency? Mfer if my PyTorch code is shitty, I’ll just buy more GPUs”
3
2
70
10,013
Arvind Sundararajan retweeted
More on H1-B: I've addressed this 18 y ago in The Black Swan: America's edge is in risk taking, not education. Any idiot can have education; you need to learn to fail. The American system has been based on trial and error. Which is why foreigners succeed here not at home.
223
1,363
230
8,231
1,308,566
Arvind Sundararajan retweeted
Improving LLM Performance - Open-Sourcing Datasets And A New Merged / Stacked LLM We are excited to announce several open-source AI contributions. MetaMathFewShot - open-source LLMs don't perform well on math and reasoning tasks. MetaMath is a popular dataset used to fine-tune open-source LLMs and improve performance. We introduce a brand new dataset, MetaMathFewShot, on HF. MetaMathFewshot has a random number of QA pairs in the prompt before being given the target question. Training on this dataset allows the model to understand the concept of few-shot prompting, a common usage pattern for end users interacting with LLMs We created 2 fine-tunes based on this dataset - Fewshot-Metamath-Mistral and Fewshot-Metamath-OrcaVicuna-Mistral. These are fine-tuned on the 7B Mistral open-source model and improvements. These fine tunes perform better than any other non-merged model on key benchmarks - Truthful QA is 53.23, Winograde is 78.45, and GSM8K 69.14 Stacking and Merging Of LLMs Stacking involves taking the layers of one model and ‘gluing’ them together with the layers of another model. For example, take layers 1-20 of a 30-layer model A and concatenate them to layers 10-30 of a 30-layer model B; your final model will have 40 layers, 20 from each original model. Merging is even simpler – take the weights of two (or more) separate models (with the same architecture, ideally) and combine them in some way to generate a new single set of weights. The method of combination can be as simple as just averaging the weights of the two models or can be slightly more refined, e.g., using SLERP to determine the weightings. In fact, the top 7B models on the HF Leaderboard are nearly all merges of various different models. We were inspired to try our hand at this as well and produced our own merged model that we are publishing on Hugging Face as well. (Links in our blog post below) At the time the model was submitted to the leaderboard, this was the second-highest-scoring 7B model on the list! We did a lot of work on Mistral 7B in this iteration; for the next one, we will tackle bigger models. The goal will be to create more open-source models that perform well in real-world scenarios. We are releasing all the weights and the datasets on HF with the hope of accelerating open-source AI even more. A lot more information is on our blog post and our HF repo. - blog.abacus.ai/blog/2024/01/…
5
54
3
222
71,983
Arvind Sundararajan retweeted
Have you tried the prompt “now use common sense” lol
113
72
15
1,117
147,130
Arvind Sundararajan retweeted
Why You Need GPT-4 For GPT-5 The MSFT-OpenAI drama is mainly about one thing - access to GPT-4 at scale. Here's why? LLM performance scales with data and compute. Many players have a lot of compute, and it is well known that data is the limiting factor when it comes to training super-powerful LLMs. The challenge, then, is to overcome the data limitation. For GPT-5 to be way more performant than GPT-4 and be good at code gen, logical reasoning, etc, it needs a massive amount of data! So, where do you get this data? You get GPT-4 to generate it! Even better, you ask it to create 10-100x its own logs! Take ChatGPT logs, give them as examples, and use the entire internet as the retrieval knowledgebase and ask it to create 10-100x more examples. You can also do this in a domain-specific way as well - create a million logical reasoning questions or math problems or whatever The more data you generate with few shot learning, prompt engineering, and other techniques, the better your LLM. GPT-4 logs are only accessible to OpenAI (via ChatGPT) and MSFT (via Bing). In addition to these logs, you need access to GPT-4 at scale - tons of compute needed to 10-100x the data! MSFT does have some disadvantages over OpenAI. - Bing chat logs are probably not as rich or as massive as ChatGPT's logs - No access to the OpenAI code base - No access to the datasets that GPT-4 was trained on OTOH OAI researchers definitely have figured out many, many tricks by now in their attempts to train GPT-5. So whoever has the employees clearly has an immense advantage. So why all the hoopla about GPT-5? A much better version of GPT-5 means far less hallucination. Near-zero hallucination translates to autonomous LLMs that can be deployed on various problems from legal advice to code generation. GPT-5 is likely way better at logical reasoning. You can achieve this by creating and training on custom datasets. An LLM that is great at retrieval and reasoning and doesn't hallucinate much is EXTREMELY VALUABLE. Would you call that "AGI"? Maybe not... but does it matter when it has superhuman retrieval and human-level reasoning skills? Unless someone else - Google, Anthropic, Open Source - catches up to GPT-4, it will be hard to create GPT-5 or equivalent. TLDR; OpenAI and to some extent, MSFT, are both in a unique position to create the next-gen LLM that some will call "AGI"
40
71
12
374
127,649
Arvind Sundararajan retweeted
Open Source AI Dance Party at The Temple Night Club!! Bollywood and EDM music playing, free food, and free drinks! We will be streaming it live on YouTube, so you can party with us wherever in the world you are!! Come celebrate Open Source AI, meet other builders, and have a great time. The venue can fit only 800 people, so RSVP quickly!! RSVP here - eventbrite.com/e/dance-party…
8
10
9
54
33,786
Arvind Sundararajan retweeted
Create Your Own Custom LLM ChatBot - An AI Brain for your organization A step by step tutorial on how to choose the best LLM and infrastructure to create a Custom ChatGPT and supercharge your business! A large language model (LLM) trained on your company's data can function like a AI brain for your business - boosting employee productivity, improving decision making and surfacing customs insights from your data Here are the key factors to consider before getting started Security - For a lot of organization, data can't leave their VPC or their cloud environments Use-cases -You might want to build a chat bot on your unstructured data like your internal wiki and sharepoint docs or a chatbot that understands all your databases. It's easy to build multiple chatbots based on your use-cases Data sources - You want to ideally connect all your important data sources including internal wikis, share point folders, contracts and even database Best LLM - There are several LLM APIs in the market. We have found different LLMs work best based on your specific problem. Try out all the LLM options and choose the best one that works for your dataset and use-case. Let's pick a classic use-case - let's say we a bunch of contracts and we want to build a contracts database that answer all the employee questions about any specific contract. There are several steps involved in building this custom chatbot. Chunking - The first step is to chunk the documents into small pieces. Chunk size and overlap are important choices based on the context length of your LLMs, type of questions being asked and how the answers are distributed in the doc Embedding - This is the process by which you convert text into numerical matrices called vectors, so we can perform operations on them Document Retrievers (Vector DBs) - We then use a vector database to store these vectors and perform a similarity search and retrieve the relevant chunks every time someone asks the chatbot a question. Context Generation Strategy - So the fundamental reason behind all these steps is that all the commercially available LLM APIs can only digest a limited number of words/tokens in a single call. This is called the context length of the LLM. So we have to know to feed the LLM relevant chunks of your data, to generate responses. Based on whether you expect long threaded chats or smaller ones, you can experiment with different techniques here Choice of LLMs To optimize your solution, it is recommended to experiment with different LLMs and determine which one delivers the best results. You can do this on your own or you can use a platform like Abacus.AI that offers a range of LLM options, including GPT3.5, GPT4, Palm, Azure OpenAI, Claude, Llama2, as well as Abacus's proprietary LLMs. Additionally, you can fine-tune an LLM on your data and use it in the Retrieval Augmented Generation technique, enjoying the best of both worlds Evaluation of Response: This is the final and most important piece. You want to prepare a set of questions and expected answers, get each of the LLMs to generate answers and evaluate the predicted answers against the expected answers. There are several techniques to do this and the Abacus AI platform automates the evaluation for you Deployment and Pipelines Once you have choosen which LLM works best, you can now deploy that LLM and set up pipelines. A pipeline automatically picks up new data (contracts in this case), chunks them, generates embeddings and fine-tunes new LLMs as needed. This way the system is now on auto-pilot Monitoring: This is the final piece. Once your AI brain is set up, you will want to evaluate it on a regular basis and and see if the performance is stable. You can now set up evaluation pipelines to ensure performance doesn't deteriorate That's it! Your Custom ChatGPT is ready to go and any of your employees can use it any time to have conversations much like they do on ChatGPT. Using a end-to-end platform like Abacus AI also gets you a fancy chat interface very much like the one on Bard or the publicly available ChatGPT where you can have threaded conversations, conversation history and provide feedback. You can also integrate this chat bot into slack or teams and ask questions that way. Resources - More detailed blog post - blog.abacus.ai/blog/2023/08/…
56
507
9
1,975
385,404
Arvind Sundararajan retweeted
friendly reminder to everyone that there isn't yet a good & proper systematic blind eval/benchmark of LLMs yet, especially those on real world data/use-cases. if i were in academia this is something i'll work on immediately.
8
23
4
232
67,452
Arvind Sundararajan retweeted
we though we wanted flying cars and not 140/280 characters, but really we wanted 32000 tokens
450
528
82
7,718
1,338,568
Arvind Sundararajan retweeted
If SVB were to be purchased, Silicon Valley will stand by the new acquirer When push comes to shove, we will stand together and support this great institution They supported us for decades! It’s time for us to do the same!
23
8
2
119
31,867
Arvind Sundararajan retweeted
Our free hands-on NLP workshop will help you become proficient in common NLP use-cases Learn how to fine-tune transformers or use micro-service APIs to solve complete business use-cases stateoftheartconference.ai/w…
7
36
Arvind Sundararajan retweeted
The church of the 95% confidence interval
33
246
28
4,319
Arvind Sundararajan retweeted
Neural networks really, truly learn. It's not a fancy trick. This is one of the most remarkable things humans have ever figured out, and the implications are difficult to overstate.
109
401
74
4,060
Arvind Sundararajan retweeted
DLOps is a thing! We are super pumped to launching the first real-time deep-learning operations platform! Building complex AI systems including NLP and recommender engines is now WAY easier :) 👇👇👇 zdnet.com/article/ai-startup…
1
22
2
154
Arvind Sundararajan retweeted
Merely knowing that a different, better way is possible, causes the breakthrough to manifest into existence. Our beliefs determine our outcomes.
7
55
3
577
Arvind Sundararajan retweeted
Move faster. Slowness anywhere justifies slowness everywhere. 2021 instead of 2022. This week instead of next week. Today instead of tomorrow. Moving fast compounds so much more than people realize.
327
3,419
547
19,508
Arvind Sundararajan retweeted
Who you work with is way more important than where you work. Great investors make a HUGE IMPACT on a start-up, especially in the early stages. In that regard, we are extremely fortunate to have the legendary Mike Volpi on our board!
3
3
34
Arvind Sundararajan retweeted
How to become expert at thing: 1 iteratively take on concrete projects and accomplish them depth wise, learning “on demand” (ie don’t learn bottom up breadth wise) 2 teach/summarize everything you learn in your own words 3 only compare yourself to younger you, never to others
180
2,767
446
14,682
Arvind Sundararajan retweeted
We are presenting "Encodings for Neural Architecture Search (NAS)" as a spotlight paper at NeurIps 2020 This is a foundational contribution to the AutoML - AI building AI - space. Thrilled to be doing it as a start-up! Details on our blog 👇 abacus.ai/blog/2020/10/02/a-…
3
13
43
Arvind Sundararajan retweeted
We just got 2 paper accepts from NeurIPs and one of them will be a spotlight paper! Super excited about the work we are doing around Neural Architecture Search and Post-hoc Debiasing of NNs! Extremely proud of the team!
6
3
66