AI Planet is proud to unveil Buddhi (Buddhi-128k-Chat-7b), one of the first few open-source chat models equipped with a 128K long context window. This model has been meticulously fine-tuned on the Mistral 7B Instruct and optimized to manage an extended context length of up to 128,000 tokens, thanks to the innovative YaRN (Yet another Rope Extension) Technique.
This breakthrough enables Buddhi to achieve a deeper understanding of context in lengthy documents or conversations, making it skilled at tasks that demand extensive context retention, memory, and recall, as well as following instructions accurately in an agentive setup.
Architecture
The Buddhi-128K-Chat model is fine-tuned on the Mistral-7B Instruct base model. We selected the Mistral 7B Instruct v0.2 as the parent model due to its superior reasoning capabilities. The architecture of the Mistral-7B model includes features like Grouped-Query Attention and Byte-fallback BPE tokenizer. Originally, this model had 32,768 maximum position embeddings. To increase the context size to 128K, we needed to modify the positional embeddings, which is where YaRN comes into play.

In our approach, we utilized the NTK-aware technique, which recommends alternative interpolation techniques for positional interpolation. One experimentation involved Dynamic-YARN, suggesting the dynamic value of the ‘s’ scale factor. This is because during inference, the sequence length changes by 1 after every word prediction. By integrating these position embeddings with the Mistral-7B Instruct base model, we achieved the 128K model.
Additionally, we fine-tuned the model on our dataset to contribute one of the very few 128K chat-based models available in the open-source community with greater reasoning capabilities than all of them.
Dataset
For the construction of our AI model, the dataset employed encompasses three distinct sections, meticulously assembled to cater specifically to chat model requirements. The initial segment draws upon Stack Exchange, presenting data as question-and-answer pairs. To enhance readability and adaptability for chat applications, we utilized the Mistral model to refine the answer formatting within this segment.
The latter two parts adopt the Alpaca formatting style, focusing on generating a context-driven conversational framework. These sections utilize the PG19 dataset, sourced from HuggingFace, as the foundational context. The distinguishing factor between them lies in the source of the question-answer pairs: while one part employs GPT-3 for generating these pairs, the other leverages GPT-4. This strategic composition of the dataset ensures a comprehensive and diverse conversational scope, tailored to optimize the chat model’s performance across a wide array of dialogue scenarios.
Benchmarks
Buddhi outperforms or matches other models in its size (128-K) class.
Short Context
The shorter context benchmarking usually involves HellaSwag, Arc challenge, MMLU, TruthFulQA, Winogrande benchmarks and so on. The below benchmarked metrics is directly fetched from Open LLM leaderboard.
Benchmarking done from Open LLM Leaderboard

Benchmarking done by our team using LM-Evaluation Harness
We use MMLU (5-shot), Winogrande (5-shot), Arc(25-shot), HellaSwag (10-shot) and TruthfulQA (0-shot) to benchmark the models using LM-Evaluation-harness library.

Long Context — Banking77

‘1R/2K’ signifies completing a single pass through all Banking77 Dataset labels, with prompts averaging 2,000 tokens each.
Currently, for long context, we benchmarked Buddhi on Banking77, we will soon release the benchmarking on DialougeRE and other metrics.
Use Cases
Large context models have a wide range of use cases right from advanced summarization capabilities to improved coherence in long text generation. Mostly, we see more use cases coming up in the autonomous agent world.
Better Memory and Recall: Small context models might forget important details mentioned earlier in a conversation. 128k models can remember a much larger chunk of the interaction history, allowing them to reference past conversations, topics, or decisions made. This leads to more coherent and natural dialogues.
Following Instructions Accurately in Agentic Setup: Imagine a user giving an agent a multi-step instruction. A small context model might miss key details later on. 128k models can retain the entire instruction, allowing them to follow through with all the steps accurately.
Higher Efficiency in Workflow Automation: For workflows that involve processing large sets of documents or data (e.g., legal document review, medical records analysis), larger context models can streamline operations by handling more content in a single pass, reducing the need for segmentation and reassembly.
Reduced Fragmentation in Text Processing, Improved Coherence in Long Text Generation & Enhanced Comprehension of Context: Smaller models often need to divide large texts into chunks, process each separately, and then attempt to integrate the results. This can lead to loss of context and coherence. Larger context models can process these texts as a whole, leading to more accurate and cohesive outcomes. In addition, it helps in Enhanced Comprehension of Context and Improved Coherence in Long Text Generation.
Deeper Analysis and Insight Generation: In fields requiring deep text analysis, such as legal research, academic studies, or market analysis, larger context models can grasp the full scope of a document. Large context models enable them to identify nuances, patterns, and connections that might be missed when analysing text in smaller segments.
Inference
Hardware requirements:
For 128k Context Length (80GB VRAM — A100 Preferred)
For 32k Context Length (40GB VRAM — A100 Preferred)
vLLM — for Faster Inference & Throughput
Considering the hardware requirements, we also conducted various experiments to optimize the throughput and inference of Buddhi-128K-Chat-7b. Specifically, we utilized vLLM, which employs Paged Attention to optimize the memory footprint during the inference of LLMs.
Try out the model on colab here: Google Colaboratory
It has various experiments that include:
Simple Prompts
Long Essay
Entire Book Summarization (86 pages)
Note: You can integrate bitsandbytes quantization to run the model on T4 GPU.
Transformers Implementation to use all 128K Context Window
If you want to experiment with the entire 128K context length and have enough compute, i.e., 80 GB VRAM with a GPU that supports Flash Attention, we have you covered with inference here as well.
Checkout HuggingFace Model Card for the code: aiplanet/buddhi-128k-chat-7b · Hugging Face
Conclusion:
While Buddhi-128K-Chat-7b and the world of large context models offer endless possibilities, they still face limitations in throughput, and latency, and have demanding compute resources due to their large sizes. Most importantly, long-context models struggle with in-context learning and may tend to fail when the complexity of tasks increases. However, with time and dedicated research by the AI community worldwide, we expect these issues will improve and the long context models will become commoditized. Moreover, we believe that this progress will also extend to support autonomous agents soon.
