Pinecone Retriever

#Overview

The PineconeRetriever class is a specialized implementation for interacting with the Pinecone vector search service, making it easier to efficiently retrieve documents based on vector similarity and metadata filters. It encapsulates the connection to the Pinecone index and the embedding model, providing methods to run text similarity searches and complex paginated queries that filter documents by specific criteria.

The main problem it solves is abstracting the complexity of manipulating Pinecone directly, making it simple to run vector searches with filters, while also ensuring proper handling of embeddings and pagination in the results. In practice, it can be used for smart search over large document bases, recommendation systems or any application that depends on efficient semantic retrieval.

For example, imagine an application that needs to find documents related to a question asked by the user, filtering by categories or other metadata – this class eases that interaction with Pinecone, taking care of everything from generating the vectors to extracting the relevant documents, without the user having to master Pinecone's query protocol.

#Execution Flow

1. Initialization of the PineconeRetriever object:
   • Receives an optional PineconeClient instance. If not provided, it internally creates a new default instance.
   • Retrieves default settings, such as batch size and vector dimension.
   • Initializes connections to the Pinecone index and the embedding model, as well as the namespace used.
2. Running a similarity search (similarity_search):
   • Receives the query text, the maximum number of results (k) and an optional metadata filter.
   • Validates the parameters, converting the query text into an embedding vector using the associated model.
   • Builds the filter query for Pinecone, compatible with search by equality or by multiple values ($eq and $in).
   • Runs the query on the index, requesting metadata for each result.
   • Formats the returned documents, removing the "text" field from the metadata and computing the similarity score.
3. Retrieving documents by metadata (get_all_docs_by_metadata):
   • Allows searching for all documents that have a given value or list of values in a metadata key, handling large volumes through automatic pagination.
   • Defines a dummy vector (zero vector) to run the query, since the focus is filtering by metadata.
   • Iterates over the result pages until there are no more pagination tokens, aggregating the documents.
   • Returns the full list of documents found, each with id, metadata and score.

#Class Methods Table

#Environment Variables

No environment variable is used directly in this class.

#Key Architecture Points and Insights

• Using the Dependency Injection pattern by allowing an external PineconeClient to be received makes testing and reuse of the Pinecone connection easier.
• Internally it applies strong encapsulation, hiding the details of configuration, embedding generation and the Pinecone query protocol.
• Careful exception handling in all methods ensures errors are easily identified and properly reported.
• Supports filters in the Pinecone format ($eq and $in), which widens the expressiveness of searches without increasing the user's complexity.
• Implements explicit pagination in the metadata query, making it possible to work with very large indexes without running out of memory.
• Uses an embedding operation for text queries that is offloaded to the client, keeping the focus on orchestrating the flow.
• The class depends on others: PineconeClient, PineconeVectorStoreConfig and ApplicationTracing, following a modular design.

#Class and Methods Description

#Class PineconeRetriever

Description

Class that manages the interaction with the Pinecone service to run vector searches based on text embeddings and metadata filters. It provides methods for simple similarity search and paginated retrieval of documents by metadata, using customizable settings for batch size, vector dimension and namespace.

Constructor Arguments

Methods

#1. __init__

Description

Initializes the PineconeRetriever object, configuring the Pinecone client, the vector configuration (dimension and batch size), the index and the namespace for queries.

Arguments

• client (Optional[PineconeClient]): Custom Pinecone client.
• If not provided, creates a new default client.

Returns

• Does not return a value.

Raises

• RuntimeError: If initialization fails because of any exception.

Examples

retriever = PineconeRetriever()  # Cria retriever com configuração padrão
custom_client = PineconeClient(...)
retriever_custom = PineconeRetriever(custom_client)  # Usa client personalizado

#2. similarity_search

Description

Runs a vector similarity search based on a text query, returning the k most relevant documents. It accepts metadata filters to refine the results.

Arguments

• query (str): Query text for the vector search.
• k (int): Maximum number of results to return.
• filter_search (Optional[Dict[str, Any]]): Dictionary with the metadata filter (e.g. {"category": ["finance","tech"]}).

Returns

• List[Dict[str, Any]]: List of documents containing id, text, metadata (without the text field) and score (similarity).

Raises

• ValueError: If query is empty or k is less than or equal to zero.
• ValueError: If the filter_search filter is invalid.
• RuntimeError: If an error occurs in embedding generation or in the Pinecone query.

Examples

results = retriever.similarity_search(
    query="How to bake a cake?",
    k=3,
    filter_search={"category": "recipes"}
)
for doc in results:
    print(doc["id"], doc["score"])

#3. get_all_docs_by_metadata

Description

Gets all documents in the Pinecone index that contain a specific value or set of values for a metadata key, using pagination to handle large volumes.

Arguments

• batch_size (int | None): Number of documents per page (configured default).
• dimension (int | None): Dimension of the dummy vector used in the query (configured default).
• target_key (str): Metadata key for the filter (e.g. "file_id").
• target_value (Union[str, List[str]]): Value or list of values to look for in the key.

Returns

• List[Dict[str, Any]]: List of documents containing id, metadata and score.

Raises

• ValueError: If target_value is not provided.
• RuntimeError: If the Pinecone query fails.

Examples

all_pdfs = retriever.get_all_docs_by_metadata(
    batch_size=20,
    target_key="file_extension",
    target_value="pdf"
)
print(f"Found {len(all_pdfs)} PDF documents")

#Usage

if __name__ == "__main__":
    import json

    pine_client = PineconeClient(
        index_name="backai-vectorstore",
        main_namespace="betterai-embeddings-dev",
    )

    retriver = PineconeRetriever(pine_client)

    # Similarity search
    similarity_results = retriver.similarity_search(
        query="What is the capital of France?",
        k=5
    )

    print("Similarity Search Results:")
    print(json.dumps(similarity_results, indent=2))

    # Metadata search
    metadata_results = retriver.get_all_docs_by_metadata(
        target_key="file_extension",
        target_value="pdf",
        batch_size=10
    )

    print("\nMetadata Search Results:")
    print(json.dumps(metadata_results, indent=2))

# python -m src.vector_store.pinecone.retriever

This documentation aims to make it easier to understand and use the PineconeRetriever class in projects that need vector searches integrated with Pinecone, with practical examples and essential details for seamless integration.