Scribd
Upload
6 views5 pages

Wizer Training Module Config

The document proposes replacing the Vanna training module with a new system due to its limitations in reliability, scalability, performance, and size. It outlines a plan for creating a semantic layer and details the methodology for embedding and performance analysis. Next steps include finalizing model selections, developing embedding functions, and conducting performance testing.

Uploaded by

akius
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
6 views5 pages

Wizer Training Module Config

The document proposes replacing the Vanna training module with a new system due to its limitations in reliability, scalability, performance, and size. It outlines a plan for creating a semantic layer and details the methodology for embedding and performance analysis. Next steps include finalizing model selections, developing embedding functions, and conducting performance testing.

Uploaded by

akius
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 5

Wizer Training Module Configuration

Document
Introduction
This document outlines the rationale for replacing the Vanna training module with a
new, more efficient system, describes the proposed approach for creating a semantic
layer, and details the methodology for embedding, performance analysis, and next
steps.

Current Implementation

Current Implementation Limitations

Why Replace the Vanna Training Module?

● Reliability: Vanna is prone to sporadic downtimes and performance


inconsistencies.
● Scalability: It's challenging to scale Vanna for enterprise-level needs due to its
excessive abstraction.
● Performance: Vanna's response generation is slow, primarily due to the Large
Language Model's lengthy processing times.
● Size: The Vanna package is overly large (>300 MB), making it unsuitable for
serverless API deployment.
● Accuracy and Code Complexity: There is a need for more complex query
generation using specialized models.
● Inference from Training Data: The current model requires extensive key-value
pairs for training, hindering rapid deployment.
Proposed Approach
Semantic Layer Creation:

Semantic Layer Creation

Classes

● WizerStore: Handles CRUD operations with the vector database. Methods


implemented for Pinecone Chroma and Qdrant include:
○ Adding DDL, documentation, and question-SQL data.
○ Retrieving related DDL, documentation, and similar question-SQL.
○ Managing training data.
def add_ddl(self, ddl: str, **kwargs) -> str:
# Implement here

def add_documentation(self, doc: str, **kwargs) -> str:


# Implement here

def add_question_sql(self, question: str, sql: str, **kwargs) -> str:


# Implement here

def get_related_ddl(self, question: str, **kwargs) -> list:


# Implement here

def get_related_documentation(self, question: str, **kwargs) -> list:


# Implement here

def get_similar_question_sql(self, question: str, **kwargs) -> list:


# Implement here

def get_training_data(self, **kwargs) -> pd.DataFrame:


# Implement here

def remove_training_data(self, id: str, **kwargs) -> bool:


# Implement here

● WizerLLM: Interfaces with the LLM. Implemented methods with Groq for
submitting prompts.

def submit_prompt(self, prompt, **kwargs) -> str:


# Implement here
End to End Flow:
Embeddings

● Selection of the optimal embedding model for performance and accuracy.


● Local storage of embedding models to avoid downloading large files for each
call.

Performance Analysis

Key Parameters:

● Speed: Benchmark against Vanna to identify bottlenecks.


● Accuracy: Research and testing with different embedding models.
● Complexity: Evaluate the complexity of generated code and model capacity.
● Inference Ability: Assess models' ability to understand context with minimal
training.

Key Goals (Levels of importance 5 = Very high, 1 = very low)

1. Identify the best model for our task (Importance: 5).


2. Determine the point of diminishing returns for model upgrades (Importance:
3).
3. Best training techniques for each LLM (Importance: 5).
4. Continuous testing and discovery of results (Importance: 5).
5. Enhance LLM accuracy over time (Importance: 4).
6. Manage training data at scale (Importance: 4).

Next Steps
● Finalize the selection of embedding and LLM models.
● Develop and implement the embedding function for documentation and
question-SQL data.
● Conduct extensive testing on performance metrics outlined above.
● Plan for scaling the training data management process.

You might also like