It does seem that the future will be one where Generative Apps will become more model (LLM) agnostic and model migration will take place; with models becoming a utility. Blue oceans are turning into red oceans very fast; and a myriad of applications and products are at threat due to developments like the expansion of LangSmith. The ecosystem is still very nascent and major changes are bound to happen. It does seem that application developers do not want to offload functionality to LLMs, for a blackbox approach. And hence including the complexity rather in the prompt/pipeline phase instead of just leveraging LLM context windows and model capabilities. LangChain, together with Haystack, are taking the lead in how the Generative App landscape is unfolding; from a open-source perspective. It will not be surprising if LangSmith will sprawl into some form of application development (and not only application management) with Pipeline and Prompt Chaining options. A graphic approach to prompt testing, like ChainForge, will also make sense. Having a tool which assist with chunking data for vector store / semantic search / RAG implementations, being able to test and tweak chunks will be of great value. Added to the main categories of Use Cases, Type, Language and Models; could be a new category of prompt techniques. Which could consider techniques like, as shown in the article below. #LargeLanguageModels #PromptEngineering #ConversationalAI @langchain @hwchase17 cobusgreyling.medium.com/lan…

A recent study found that when LLMs are presented with longer input, LLM performance is best when relevant content is at the start or end of the input context. Performance degrades when relevant information is in the middle of long context. A few days ago Haystack by deepset released a component which optimises the layout of selected documents in the LLM context window. The component is a way to work around the problem identified in the paper. Why I particularly like this implementation from Haystack, is the fact that it's a good example of how innovation in the pre-LLM functionality, or the pipeline phase, can remedy inherent vulnerabilities of a LLM. Thank you @tuanacelik for telling me about this functionality and technical assistance. 🙂 @deepset_ai Read more here: cobusgreyling.medium.com/aha…

Large Language Models (LLMs) are known to hallucinate. Hallucination is when a LLM generates a highly succinct and highly plausible answer; but factually incorrect. Hallucination can be negated by injecting prompts with contextually relevant data which the LLM can reference. Growing LLM context size has the allure that large swaths of contextual reference data can merely be submitted to the LLM to act as reference data. Reference data which will create a contextual reference for the LLM and in turn negate hallucination… A recent study found that LLMs perform better when the relevant information is located at the beginning or end of the input context. However, when relevant context is in the middle of longer contexts, the retrieval performance is degraded considerably. This is also the case for models specifically designed for long contexts. Read more in the article below. #LargeLanguageModels #PromptEngineering #LLMs cobusgreyling.medium.com/doe…

This article considers how Ragas can be combined with LangSmith for more detailed insights into how Ragas goes about evaluating a RAG/LLM implementation. Currently Ragas makes use of OpenAI, but it would make sense for Ragas to become more LLM agnostic; And Ragas is based on LangChain. The LangSmith integration lifts the veil on cost and latency running Ragas. For this assessment based implementation, latency might not be that crucial, but cost can be a challenge. In short, what is Ragas? Ragas is a framework which can be used to test and evaluate RAG implementations. As seen in the image below, the RAG evaluation process divides the assessment into to categories, Generation and Retrieval. Generation is again divided into two metrics, faithfulness and relevance. Retrieval can be divided into Context Precision and Context Recall. Link to the full article: cobusgreyling.medium.com/com… #LangChain #LargeLanguageModels #PromptEngineering @langchain @hwchase17

Steps In Evaluating Retrieval Augmented Generation (RAG) Pipelines - The basic principle of RAG is to leverage external data sources. For each user query or question, a contextual chunk of text is retrieved to inject into the prompt. This chunk of text is retrieved based on its semantic similarity with the user question. But how can a RAG implementation be tested and benchmarked over time? Input: Question:  - These are the questions the RAG pipeline will be evaluated on. Answer:  - The answer generated from the RAG pipeline which will be presented to the user. Contexts:  - The contexts which will be passed into the LLM to answer the question. Ground Truths:  - The ground truth answer to the questions. Output: Faithfulness, Answer Relevancy, Context Relevancy and Context Recall. Link to the full article: medium.com/@cobusgreyling/st… #LargeLanguageModels #PromptEngineering #RAG

How to Mitigate LLM Hallucination and Single LLM Vendor Dependancy (Link to the full article in the comments) Four years ago I wrote about the importance of context when developing a chatbot. Context is more relevant now with LLMs than ever before. Injecting prompt with a contextual reference (RAG) at inference does two things. (1) It negates hallucination, (2) it vastly improves the performance of LLMs. I have read many comments that the models hosted on HuggingFace is just not as accurate and coherent as OpenAI's models...but have you tried interacting with these model with a contextual reference? I believe a contextual reference greatly enhances the performance of LLMs. In the two examples below the model google/flan-t5-xxl, hosted by HuggingFace, is used in both instances on the right and left. On the left, no RAG or contextual reference approach is taken. However, on the right the prompt is given an a correct answer is given by the LLM. #LargeLanguageModels #promptEngineering #LLMs cobusgreyling.medium.com/mit…

The graph below graphically illustrates how the accuracy improves at the beginning and end of the information entered. And the performance deprecation when referencing data in the middle is also visible.

How Does Large Language Models Use Long Contexts? And how to manage the performance and cost of large context input to LLMs? - There is a deprecation in LLM performance when large context windows are leveraged. - Offloading complexity to the LLM provider will turn the application into a black-box without the ability to granularly manage cost, input and output token use, model performance and context. - A simplistic approach will incur technical debt which will have to be addressed later in the application lifecycle. - Offloading complexity and data management to the LLM also closely ties the Generative App to a specific LLM. Generative Apps can be LLM agnostic by following a RAG approach. - The ideal scenario is where the LLM is a utility and do not manage data or hold application complexity. - Via a RAG implementation, use-cases demanding large context windows can be managed outside the ambit of the LLM. Full post: cobusgreyling.medium.com/how…

Lately I've started to consider the growing context of various Large Language Models (LLMs) to what extent it can be used and how a principle like RAG applies. There is a split taking place in LLM usage into two main streams; commercial single end-users & enterprise implementations. For end-users there are polished UIs available like ChatGPT, HuggingChat…Cohere has also launched a personal assistant called Coral. There are also growing playground functionality from companies like Vellum and Vercel. All of these User Interfaces are no-code web-based UIs which allows for individuals to use LLMs as productivity tools. But, the chasm between personal use and enterprise use is huge. And while LLM suppliers are aiming for end-users as part of their product offering…the challenge and complexity lies with enterprise implementations. Link to the full article: cobusgreyling.medium.com/rag… #LargeLanguageModels #PromptEngineering #ConversationalAI

In upcoming articles I will be focussing on the available and most efficient methods of presenting LLMs with contextual reference data at inference. In the article below I consider a RAG implementation via Haystack by deepset which accesses a LLM and prepared data via Hugging Face. Chunking data is a method of segmented text data into contextually coherent text segments of about 200 words each. These chunks of text is injected into the prompt, and presented with instructions, to the LLM at inference. The article breaks the process down into its most basic form and clearly illustrates how you can build an easy prototype version. @deepset_ai @huggingface #LargeLanguageModels #PromptEngineer #ConversationUI medium.com/@cobusgreyling/ch…

Considering the Venn Diagram below, when thinking of the LLM use case in general, three main considerations are shown, and their relationships to each-other. - User Input Context / User Intent Any user input will have a certain context. The context is the frame of reference of the user, which most probably informs their query. In traditional chatbot lingo, this is the intent of the user; the intention which moved the user to interact with the system. - LLM Knowledge This is the knowledge which is baked into the LLM. hence the knowledge with which the LLM is trained, which have a definite cut-off in terms of recency and current affairs. The LLM has the ability to act as the backbone of the conversational dialog management and formulate succinct responses via natural language generation (NLG). - External Contextual Reference Data This is the supplementary data which are retrieved in chunks; semantic search and embeddings are used to ensure contextually relevant data is returned from the available corpus. To read more: medium.com/@cobusgreyling/ra… #LargeLanguageModels #PromptEngineering #ConversationalAI

Two LLM Based Autonomous Agents Debate Each Other! This working code example using the LangChain framework illustrates how two agents can debate each other, after each agent has been assigned a persona and an objective. The agents have access to tools which they can leverage for their response. As you will see below, after examining the LangSmith trace, the LLM forms the backbone of the application and the tools provide relevant and up-to-date information. The one agent is named "AI Accelerationist" and the other "AI Alarmist". Full Article: cobusgreyling.medium.com/two… #LargeLanguageModels #PromptEngineering #ConversationalAI

Of late I have been spending most of my time on the LLM and Generative AI tooling landscape. However, I tried to update the Conversational AI technologies landscape today. There has been additions to the "LLM enabled chatbot development frameworks", voicebot and chatbot frameworks. This past week Vellum announced their entry into the flow-builder / prompt-chaining market. I cannot help but ask myself to what extent the traditional chatbot vendors see this as a threat? It does seem like the chatbot/voicebot focus is CCAI use-cases, Voicebots and LLM-assisted development. Any insights from the community will be greatly appreciated. 🙂 #Chatbots #Voicebots #ConversationalAI More on Medium: cobusgreyling.medium.com/the…

As I always say, the complexity of any implementation needs to be accommodated somewhere. In the case of LLMs, the model can be fine-tuned, or advanced prompting can be used in an autonomous agent or via prompt-chaining. In the last two examples mentioned, the complexity is absorbed in prompt engineering and not via LLM fine-tuning. The objective of contextual iterative prompting is to absorb the complexity demanded by a specific implementation; and not offload any model fine-tuning to the LLM. Contextual iterative prompting is not a new approach, what this paper considers is the creation and automation of an iterative Context-Aware Prompter. At each step (each dialog turn) the Prompter learns to process the query and previously gathered evidence, and composes a prompt which steers the LLM to recall the next piece of knowledge. Read more here: ⬇️ #LargeLanguageModels #PromptEngineering #ConversationalAI cobusgreyling.medium.com/ite…

There were a few market shifts and trends since the previous LLM Stack chart, below are a few considerations: (The link the the full article is in the comments.) 1. There has been a diversification in LLM models available. LLM suppliers like OpenAI, Cohere and others have multiple models and these models all have particular strengths and target implementations. 2. Default functionality supplied by LLM providers are expanding, especially in the area of structure and standard built-in playground functionality. These LLM developments in some cases supersedes existing products. Consider here the growing LLM context windows, applications needing to reference multiple models and more. I would argue products higher up in the stack are more vulnerable. 3. LLMs are releasing end-user chat interfaces, like HuggingChat, ChatGPT and Coral from Cohere. Again, challenging the differentiation and propriety software and intellectual property of each offering. 4. Prompt Engineering techniques are growing and the flexibility and utility of LLMs are really coming to the fore. 5. Fine-Tuning of LLMs are still relevant to create industry and market related models, for instance medical, legal, engineering, etc. 6. However, for general implementations a popular architecture is one of RAG, where accurate and contextually relevant reference data is included in the prompt to increase LLM response accuracy and negates hallucination. 7. Embeddings, Vector Stores and the notion of semantic search and data retrieval is also becoming more important. #LargeLanguageModels #PromptEngineering #ConversationalAI Read more here: cobusgreyling.medium.com/fou…

Emerging Large Language Model (LLM) Application Architecture Due to the highly unstructured nature of Large Language Models (LLMs), there are thought and market shifts taking place on how to implement LLMs. Due to the unstructured nature of human language, the input to LLMs are conversational and unstructured, in the form of Prompt Engineering. And the output of LLMs is also conversational and unstructured; a highly succinct form of natural language generation (NLG). LLMs introduced functionality to fine-tune and create custom models. And an initial approach to customising LLMs was creating custom models via fine-tuning. This approach has fallen into disfavour for three reasons: 1. As LLMs have both a generative and predictive side. The generative power of LLMs is easier to leverage than the predictive power. If the generative side of LLMs are presented with contextual, concise and relevant data at inference-time, hallucination is negated. 2. Fine-tuning LLMs involves training data curation, transformation and cost. Fine-tuned models are frozen with a definite time-stamp and will still demand innovation around prompt creation and data presentation to the LLM. 3. When classifying text based on pre-defined classes or intents, NLUstill has an advantage with built-in efficiencies. The aim of fine-tuning of LLMs is to engender more accurate and succinct reasoning and answers. This also solves for one of the big problems with LLMs; hallucination, where the LLM returns highly plausible but incorrect answers. Link to the full article: cobusgreyling.medium.com/eme…