Search with Re-Ranking

Bigdata.com API enables one to search through a vast amount of documents instantly. To enhance the search results even further, Bigdata provides a Cross Encoder re-ranker that assigns a new relevance score to each candidate document. This allows the system to filter out less pertinent results and focus on the most promising ones.

Introduction

In this use case, we explore the effect of cross encoder in action. In the first step, a similarity search rapidly scans hundreds of millions of documents to surface a list of matches. While this method efficiently gathers candidates, not all of them may be perfectly aligned with the query’s intent. We call this the baseline search results. Then we run search for the same query with cross encoder re-ranking enabled, which filters out documents that do not meet the relevance threshold.

Step 0: Prerequisites

We need to import the Bigdata client library with the supporting modules:

import html

from IPython.display import display, HTML
from bigdata_client import Bigdata
from bigdata_client.daterange import RollingDateRange
from bigdata_client.query import Similarity

Step 1: Initialization

First we need to initialize the Bigdata class. This class will be used to interact with the Bigdata API.

The authentication is done here too, by loading the BIGDATA_USERNAME and BIGDATA_PASSWORD environment variables. If they are not set, you can either set them or pass the username and password arguments to the Bigdata class:

bigdata = Bigdata()

Step 2: Define Helper Functions

We define a helper function to print the search results. This function prints the search results in a readable format:

def escape_special_chars(text):
    """Escapes special characters for safe HTML display."""
    text = html.escape(text)  # Escapes HTML special characters like <, >, &
    # text = text.replace(r"$", r"\$")  # Escape the dollar sign properly
    text = text.replace("  ", "&nbsp;&nbsp;")  # Preserve double spaces
    return text


def infer_ranks(idx, document):
    """Infers the original and new ranks for a document."""
    if getattr(document, "baseline_rank", None):
        new_rank = idx
        original_rank = document.baseline_rank
    else:
        new_rank = idx
        original_rank = None
    return new_rank, original_rank


def print_results_html(results):
    """Prints search results in a readable format."""
    html_output = """
    <style>
        .results-container {
            font-family: Arial, sans-serif;
            background: #1e1e1e;
            color: white;
            padding: 20px;
            border-radius: 10px;
            max-width: 800px;
            margin: auto;
            box-shadow: 0px 4px 10px rgba(0, 0, 0, 0.5);
        }
        .result-card {
            border: 1px solid #444;
            padding: 15px;
            margin: 15px 0;
            border-radius: 8px;
            background: #2a2a2a;
            transition: transform 0.2s, box-shadow 0.2s;
        }
        .result-card:hover {
            transform: scale(1.02);
            box-shadow: 0px 4px 10px rgba(255, 255, 255, 0.1);
        }
        .rank-container {
            display: flex;
            gap: 10px; /* Space between rank bubbles */
            align-items: center;
            margin-bottom: 10px;
        }
        .rank-badge {
            font-weight: bold;
            font-size: 16px;
            padding: 6px 12px;
            border-radius: 20px;
            display: inline-block;
            color: white;
        }
        .badge-blue {
            background: #1E88E5;
        }
        .badge-green {
            background: #28A745;
        }
        .headline {
            font-size: 20px;
            font-weight: bold;
            color: #ffcc00;
        }
        .timestamp {
            font-size: 14px;
            color: #cccccc;
        }
        .relevance {
            font-size: 16px;
            font-weight: bold;
            color: #00d1b2;
        }
        .text {
            font-size: 16px;
            line-height: 1.6;
            color: #dddddd;
        }
    </style>
    <div class='results-container'>
    """

    for idx, document in enumerate(results, 1):
        # Infer ranks for the document
        new_rank, original_rank = infer_ranks(idx, document)

        headline = escape_special_chars(document.headline.title())
        timestamp = document.timestamp.strftime("%Y-%m-%d %H:%M:%S")
        relevance = round(document.chunks[0].relevance, 2)
        first_chunk_text = escape_special_chars(document.chunks[0].text)

        html_output += """
        <div class='result-card'>
        """

        # Display rank badges if available
        if original_rank:
            html_output += f"""
                <div class='rank-container'>
                    <div class='rank-badge badge-green'>New Rank: {new_rank}</div>
                    <div class='rank-badge badge-blue'>Original Rank: {original_rank}</div>
                </div>
            """
        else:
            html_output += f"""
                <div class='rank-container'>
                    <div class='rank-badge badge-blue'>Rank: {new_rank}</div>
                </div>
            """

        html_output += f"""
            <div class='headline'>{headline}</div>
            <div class='timestamp'><strong>Timestamp:</strong> {timestamp}</div>
            <div class='relevance'><strong>📘 Relevance:</strong> {relevance}</div>
            <div class='text'>{first_chunk_text}</div>
        </div>
        """

    html_output += "</div>"

    display(HTML(html_output))

Step 3: Define Search Query

As an example, we explore the potential impact of President Trump’s proposed tax cuts---referred to as “Trump 2.0” tax cuts---on the federal deficit. We use a Similarity query to search for documents that are similar to the query string.

query = Similarity('Trump 2.0 tax cuts impact on federal deficit')

Step 4: Phase 1 - Baseline Search

We first run a search with the query string to see the results without cross encoder re-ranking. This will give us a baseline to compare the results later once we enable cross encoder re-ranking.

results_baseline = bigdata.search.new(
    query,
    date_range=RollingDateRange.LAST_THIRTY_DAYS,
).run(limit=10)

# Baseline search results, a.k.a. without cross encoder re-ranking enabled
print_results_html(results_baseline)

Step 4: Phase 2 - Search with Re-Ranking

The rerank_threshold argument is used to apply the re-ranking using the cross encoder between the query and the initial search results. This will filter out the documents that have a re-ranking relevance score below this threshold, so enhances the relevance of the final search results.

results_rerank = bigdata.search.new(
    query,
    rerank_threshold=0.9,  # Filter out results with relevance below 0.9
    date_range=RollingDateRange.LAST_THIRTY_DAYS,
).run(limit=10)

We compare the results of the baseline search with the re-ranked search results to see the impact of cross encoder re-ranking.

# Add baseline rank to each document in the re-ranked results
baseline_document_ids = [doc.id for doc in results_baseline]
for document in results_rerank:
    try:
        baseline_rank = baseline_document_ids.index(document.id) + 1
    except ValueError:
        baseline_rank = None
    document.baseline_rank = baseline_rank

print_results_html(results_rerank)

Conclusion

The results are re-ranked to prioritize highly relevant matches with the query intent, ensuring the most pertinent results appear at the top. For example, results related to adjacent topics (e.g., “tariffs”) are deprioritized in favor of those directly aligned with the search query. Any results falling below the rerank_threshold are filtered out, improving overall relevance.

For more details, please refer to the Bigdata.com API official documentation.

Happy Searching! 🚀