T_Frazao_assessment

Question 1 - Summary Here is a summary of the overall approach I took for training a text generation model on a small dataset of 1000 examples: Data Preparation Loaded the dataset Preprocessed the data by tokenizing the input text and target text using the T5 tokenizer Split the tokenized dataset into train (1000 examples) and validation sets Defined a collate function to pad the batches to consistent lengths Model Training Initialized a T5ForConditionalGeneration model for sequence-to-sequence generation Defined training arguments including learning rate, epochs, batch size and output steps Created a Seq2SeqTrainer to train the model on the small training set Trained for 5/10 epochs with a small batch size of 16 Evaluation and Generation Evaluated the trained model on the validation set to check loss Generated text by providing a code documentation prompt to the model Decoded the output tokens into readable text Overall, I preprocessed the data into a format suitable for seq2seq training, trained a T5 model on 1000 examples, and evaluated it's generation capabilities. The small dataset limited the model performance, but this approach enabled fast iteration and testing of the end-to-end pipeline. Question 2 - Summary I installed the genai_stack, sentence_transformers, and langchain packages. I then imported various modules from langchain and genai_stack to build an AI assistant stack. The stack includes: LangchainETL to load data from a CSV file LangchainEmbedding to generate embeddings ChromaDB as a vector database HuggingFaceModel using GPT-4all model PromptEngine LangChainRetriever ConversationBufferMemory I loaded a CSV called billionaires.csv and indexed it, then tested the retriever by asking it two questions: List the category where billionaires work Name a billionaire starting with E And printed the retrieved responses. I had difficulties running on google colab because of RAM when calling Stack(). I decided to work on jupyter notebook however I had problems when testing also with the kernel that started crashing and I couldnt find the error.