Arvind Sundararajan retweeted
More on H1-B:
I've addressed this 18 y ago in The Black Swan: America's edge is in risk taking, not education. Any idiot can have education; you need to learn to fail.
The American system has been based on trial and error. Which is why foreigners succeed here not at home.
Arvind Sundararajan retweeted
Improving LLM Performance - Open-Sourcing Datasets And A New Merged / Stacked LLM
We are excited to announce several open-source AI contributions.
MetaMathFewShot - open-source LLMs don't perform well on math and reasoning tasks. MetaMath is a popular dataset used to fine-tune open-source LLMs and improve performance. We introduce a brand new dataset, MetaMathFewShot, on HF.
MetaMathFewshot has a random number of QA pairs in the prompt before being given the target question. Training on this dataset allows the model to understand the concept of few-shot prompting, a common usage pattern for end users interacting with LLMs
We created 2 fine-tunes based on this dataset - Fewshot-Metamath-Mistral and Fewshot-Metamath-OrcaVicuna-Mistral. These are fine-tuned on the 7B Mistral open-source model and improvements.
These fine tunes perform better than any other non-merged model on key benchmarks - Truthful QA is 53.23, Winograde is 78.45, and GSM8K 69.14
Stacking and Merging Of LLMs
Stacking involves taking the layers of one model and ‘gluing’ them together with the layers of another model. For example, take layers 1-20 of a 30-layer model A and concatenate them to layers 10-30 of a 30-layer model B; your final model will have 40 layers, 20 from each original model.
Merging is even simpler – take the weights of two (or more) separate models (with the same architecture, ideally) and combine them in some way to generate a new single set of weights. The method of combination can be as simple as just averaging the weights of the two models or can be slightly more refined, e.g., using SLERP to determine the weightings.
In fact, the top 7B models on the HF Leaderboard are nearly all merges of various different models. We were inspired to try our hand at this as well and produced our own merged model that we are publishing on Hugging Face as well. (Links in our blog post below)
At the time the model was submitted to the leaderboard, this was the second-highest-scoring 7B model on the list!
We did a lot of work on Mistral 7B in this iteration; for the next one, we will tackle bigger models. The goal will be to create more open-source models that perform well in real-world scenarios.
We are releasing all the weights and the datasets on HF with the hope of accelerating open-source AI even more.
A lot more information is on our blog post and our HF repo.
- blog.abacus.ai/blog/2024/01/…
Arvind Sundararajan retweeted
Replying to @ylecun @geoffreyhinton
Have you tried the prompt “now use common sense” lol
Arvind Sundararajan retweeted
Why You Need GPT-4 For GPT-5
The MSFT-OpenAI drama is mainly about one thing - access to GPT-4 at scale.
Here's why?
LLM performance scales with data and compute. Many players have a lot of compute, and it is well known that data is the limiting factor when it comes to training super-powerful LLMs.
The challenge, then, is to overcome the data limitation. For GPT-5 to be way more performant than GPT-4 and be good at code gen, logical reasoning, etc, it needs a massive amount of data!
So, where do you get this data? You get GPT-4 to generate it! Even better, you ask it to create 10-100x its own logs!
Take ChatGPT logs, give them as examples, and use the entire internet as the retrieval knowledgebase and ask it to create 10-100x more examples. You can also do this in a domain-specific way as well - create a million logical reasoning questions or math problems or whatever
The more data you generate with few shot learning, prompt engineering, and other techniques, the better your LLM.
GPT-4 logs are only accessible to OpenAI (via ChatGPT) and MSFT (via Bing). In addition to these logs, you need access to GPT-4 at scale - tons of compute needed to 10-100x the data!
MSFT does have some disadvantages over OpenAI.
- Bing chat logs are probably not as rich or as massive as ChatGPT's logs
- No access to the OpenAI code base
- No access to the datasets that GPT-4 was trained on
OTOH OAI researchers definitely have figured out many, many tricks by now in their attempts to train GPT-5. So whoever has the employees clearly has an immense advantage.
So why all the hoopla about GPT-5? A much better version of GPT-5 means far less hallucination. Near-zero hallucination translates to autonomous LLMs that can be deployed on various problems from legal advice to code generation.
GPT-5 is likely way better at logical reasoning. You can achieve this by creating and training on custom datasets. An LLM that is great at retrieval and reasoning and doesn't hallucinate much is EXTREMELY VALUABLE.
Would you call that "AGI"? Maybe not... but does it matter when it has superhuman retrieval and human-level reasoning skills?
Unless someone else - Google, Anthropic, Open Source - catches up to GPT-4, it will be hard to create GPT-5 or equivalent.
TLDR; OpenAI and to some extent, MSFT, are both in a unique position to create the next-gen LLM that some will call "AGI"
Arvind Sundararajan retweeted
Open Source AI Dance Party at The Temple Night Club!!
Bollywood and EDM music playing, free food, and free drinks!
We will be streaming it live on YouTube, so you can party with us wherever in the world you are!!
Come celebrate Open Source AI, meet other builders, and have a great time.
The venue can fit only 800 people, so RSVP quickly!!
RSVP here - eventbrite.com/e/dance-party…
Arvind Sundararajan retweeted
Create Your Own Custom LLM ChatBot - An AI Brain for your organization
A step by step tutorial on how to choose the best LLM and infrastructure to create a Custom ChatGPT and supercharge your business!
A large language model (LLM) trained on your company's data can function like a AI brain for your business - boosting employee productivity, improving decision making and surfacing customs insights from your data
Here are the key factors to consider before getting started
Security - For a lot of organization, data can't leave their VPC or their cloud environments
Use-cases -You might want to build a chat bot on your unstructured data like your internal wiki and sharepoint docs or a chatbot that understands all your databases. It's easy to build multiple chatbots based on your use-cases
Data sources - You want to ideally connect all your important data sources including internal wikis, share point folders, contracts and even database
Best LLM - There are several LLM APIs in the market. We have found different LLMs work best based on your specific problem. Try out all the LLM options and choose the best one that works for your dataset and use-case.
Let's pick a classic use-case - let's say we a bunch of contracts and we want to build a contracts database that answer all the employee questions about any specific contract. There are several steps involved in building this custom chatbot.
Chunking - The first step is to chunk the documents into small pieces. Chunk size and overlap are important choices based on the context length of your LLMs, type of questions being asked and how the answers are distributed in the doc
Embedding - This is the process by which you convert text into numerical matrices called vectors, so we can perform operations on them
Document Retrievers (Vector DBs) - We then use a vector database to store these vectors and perform a similarity search and retrieve the relevant chunks every time someone asks the chatbot a question.
Context Generation Strategy - So the fundamental reason behind all these steps is that all the commercially available LLM APIs can only digest a limited number of words/tokens in a single call. This is called the context length of the LLM.
So we have to know to feed the LLM relevant chunks of your data, to generate responses. Based on whether you expect long threaded chats or smaller ones, you can experiment with different techniques here
Choice of LLMs To optimize your solution, it is recommended to experiment with different LLMs and determine which one delivers the best results.
You can do this on your own or you can use a platform like Abacus.AI that offers a range of LLM options, including GPT3.5, GPT4, Palm, Azure OpenAI, Claude, Llama2, as well as Abacus's proprietary LLMs.
Additionally, you can fine-tune an LLM on your data and use it in the Retrieval Augmented Generation technique, enjoying the best of both worlds
Evaluation of Response: This is the final and most important piece. You want to prepare a set of questions and expected answers, get each of the LLMs to generate answers and evaluate the predicted answers against the expected answers. There are several techniques to do this and the Abacus AI platform automates the evaluation for you
Deployment and Pipelines Once you have choosen which LLM works best, you can now deploy that LLM and set up pipelines. A pipeline automatically picks up new data (contracts in this case), chunks them, generates embeddings and fine-tunes new LLMs as needed. This way the system is now on auto-pilot
Monitoring: This is the final piece. Once your AI brain is set up, you will want to evaluate it on a regular basis and and see if the performance is stable. You can now set up evaluation pipelines to ensure performance doesn't deteriorate
That's it! Your Custom ChatGPT is ready to go and any of your employees can use it any time to have conversations much like they do on ChatGPT. Using a end-to-end platform like Abacus AI also gets you a fancy chat interface very much like the one on Bard or the publicly available ChatGPT where you can have threaded conversations, conversation history and provide feedback.
You can also integrate this chat bot into slack or teams and ask questions that way.
Resources - More detailed blog post - blog.abacus.ai/blog/2023/08/…
Arvind Sundararajan retweeted
friendly reminder to everyone that there isn't yet a good & proper systematic blind eval/benchmark of LLMs yet, especially those on real world data/use-cases.
if i were in academia this is something i'll work on immediately.
Arvind Sundararajan retweeted
we though we wanted flying cars and not 140/280 characters, but really we wanted 32000 tokens
Arvind Sundararajan retweeted
If SVB were to be purchased, Silicon Valley will stand by the new acquirer
When push comes to shove, we will stand together and support this great institution
They supported us for decades!
It’s time for us to do the same!
Arvind Sundararajan retweeted
Our free hands-on NLP workshop will help you become proficient in common NLP use-cases
Learn how to fine-tune transformers or use micro-service APIs to solve complete business use-cases
stateoftheartconference.ai/w…
Arvind Sundararajan retweeted
Neural networks really, truly learn. It's not a fancy trick.
This is one of the most remarkable things humans have ever figured out, and the implications are difficult to overstate.
Arvind Sundararajan retweeted
DLOps is a thing!
We are super pumped to launching the first real-time deep-learning operations platform!
Building complex AI systems including NLP and recommender engines is now WAY easier :)
👇👇👇
zdnet.com/article/ai-startup…
Arvind Sundararajan retweeted
Merely knowing that a different, better way is possible, causes the breakthrough to manifest into existence. Our beliefs determine our outcomes.
Arvind Sundararajan retweeted
Move faster. Slowness anywhere justifies slowness everywhere.
2021 instead of 2022. This week instead of next week. Today instead of tomorrow.
Moving fast compounds so much more than people realize.
Arvind Sundararajan retweeted
Who you work with is way more important than where you work.
Great investors make a HUGE IMPACT on a start-up, especially in the early stages.
In that regard, we are extremely fortunate to have the legendary Mike Volpi on our board!
Congratulations to @bindureddy, @arvinds, and Siddartha Naidu on today’s announcement! We are excited to continue to invest in @abacusai and support your mission to democratize AI. techcrunch.com/2020/11/18/ab…
Arvind Sundararajan retweeted
How to become expert at thing:
1 iteratively take on concrete projects and accomplish them depth wise, learning “on demand” (ie don’t learn bottom up breadth wise)
2 teach/summarize everything you learn in your own words
3 only compare yourself to younger you, never to others
Arvind Sundararajan retweeted
We are presenting "Encodings for Neural Architecture Search (NAS)" as a spotlight paper at NeurIps 2020
This is a foundational contribution to the AutoML - AI building AI - space.
Thrilled to be doing it as a start-up!
Details on our blog 👇
abacus.ai/blog/2020/10/02/a-…


















