How much time do you spend each week sifting through data?

A) Less than 2 hours

B) 2-5 hours

C) 5-10 hours

D) More than 10 hours

What is Metadata Filtering?

Technical Definition:

Metadata filtering involves enriching data records with additional attributes such as content type, author, creation date, or custom tags. This added information enables the AI to filter and sort data more effectively, providing insights that go beyond just the text. In Epsilla Cloud, metadata can be applied to different data sources like Local Files, S3, and SharePoint. The metadata is stored in a JSON object, attached to the original files, creating a richer knowledge base for the AI to leverage.

Analogy:

“Think of metadata filtering like a librarian who knows every book in the library—not just by title, but by its themes, genre, and relevance to what you’re searching for. In the same way, metadata filtering helps the AI sort through vast amounts of data in finance, legal, academic, etc, to find the most relevant pieces for your analysis.”

Mastering Metadata Filtering: Step-by-Step Guide to Building Smarter AI with Epsilla Cloud

In our last article, we discussed how to create a financial analyst AI agent using Epsilla's RAG-as-a-Service platform, guiding you through the process of building a specialized LLM agent for financial data analysis. Now, let’s explore how metadata filtering can take this agent to the next level. By incorporating metadata filters in Epsilla Cloud, you can fine-tune data retrieval, allowing your AI to provide more relevant and accurate insights. This simple addition significantly improves the efficiency of your AI Agent, making data access faster and more streamlined.

Step 1. Metadata Filtering Basics

Learn how to set up metadata filtering in Epsilla Cloud, including creating metadata files, setting up field mappings, and preparing your knowledge base for smarter data retrieval.

Step 2. Building and Customizing Your AI Agent

Now that our metadata setup is complete, let’s move on to building the AI Agent. Watch this video to see how to create the application, customize its role, and integrate the right knowledge base.

Here is the provided role for the AI agent (the system prompt in technical terms):

Step 3. Dynamic Filtering Workflow

In this video, we’ll create a dynamic filtering workflow using the LLM completion node. See how we make the AI smart enough to adjust its filters based on user questions, like focusing on Apple’s stock performance in Q3.

Here is the System Message to use in the LLM node for user query intention analysis and filter generation:

CompanyName = '{company_name}'

CompanyName = 'Apple'

(CompanyName = '{company1_name}' OR CompanyName = '{company2_name}' OR ...)

(CompanyName = 'Nvidia' OR CompanyName = 'Meta')

Time = '{the_quarter} 2024'

Time = 'Q1 2024'

(Time = '{the_quarter_1} 2024' OR Time = '{the_quarter_2} 2024' OR ...)

(Time = 'Q2 2024' OR Time = 'Q3 2024')

(CompanyName = 'Microsoft' OR CompanyName = 'Alphabet') AND Time = 'Q2 2024'

true

Step 4. Evaluating AI Performance with Metadata Filtering

Finally, let’s evaluate the difference metadata filtering makes. This video compares the accuracy of AI Agents with and without metadata filtering, showing why it’s essential for precise, relevant answers.

Question in the evaluation:

Human labeled answer:

Conclusion: Ready to Unleash the Power of Metadata Filtering?

Metadata filtering is a game-changer, turning data overload into data empowerment. With Epsilla Cloud, you can create AI assistants that don’t just find the right information—they understand its context and relevance. By leveraging semantic search, smart data tagging, and AI automation, you can transform AI agents across industries—be it finance, legal, healthcare, tax, or research—into intelligent virtual assistants tailored to your specific needs.