Visualize theme volume evolution

Overview

This guide demonstrates how to use the Bigdata Search Volume API endpoint (/v1/search/volume) to retrieve and visualize theme volume data over time. The script displays the evolution of documents, chunks, and sentiment for any given theme, helping you track narrative trends and coverage patterns.

GitHub Repository

Access the complete source code, requirements, and setup instructions.

Features

✅ Retrieves theme volume data from the Bigdata Search Volume API
✅ Visualizes three key metrics: number of documents, number of chunks, and sentiment
✅ Displays daily values as bars (documents and chunks) or lines (sentiment)
✅ Overlays weekly average trends for better pattern recognition
✅ Generates high-resolution PNG charts with theme-specific filenames
✅ Supports custom date ranges and themes via command-line arguments
✅ Optional breakdown of the volume by source (group_by)

Use Cases

The Search Volume endpoint is valuable for several scenarios:

• Evaluate Coverage: See how many unique documents Bigdata has available for your particular query
• Narrative/Thematic Screeners: Create a query and see how the defined narrative evolves over time
• Query Strategy: Check coverage and plan how to structure your queries accordingly

Efficient Usage: The /v1/search/volume endpoint consumes exactly 1 API Query Unit regardless of the time period queried (up to 60 weeks maximum), making it a very efficient way to visualize narrative evolution over time.

Flexible Filtering: The Search Volume endpoint supports all the same filters as the Search endpoint, including entity, source, category, keyword, and sentiment filters. For more details, check out the Search Volume API specification.

Group the volume by source

Add the optional group_by parameter to see which sources contribute to the volume. Pass ["source"] to break down the results:

POST /v1/search/volume
{
  "query": {
    "text": "Tariffs impact",
    "filters": {
      "timestamp": { "start": "2026-01-01T00:00:00", "end": "2026-03-31T23:59:59" }
    }
  },
  "group_by": ["source"]
}

The response keeps results.volume and results.total unchanged and adds results.groups.sources. Each row carries the source id, the number of distinct documents and the number of chunks, sorted by chunks:

{
  "results": {
    "volume": [ ... ],
    "total": { "documents": 820, "chunks": 1087, "sentiment": -0.14 },
    "groups": {
      "sources": [
        { "id": "B1A50F", "documents": 58, "chunks": 130 },
        { "id": "9635A9", "documents": 2, "chunks": 2 }
      ]
    }
  },
  "usage": { "api_query_units": 1 }
}

The source id matches the source filter, so a row can be turned straight into a follow-up query restricted to that source. The list holds at most 1,000 sources; broad queries over long windows can hit that cap, so narrow the query or the time range when you need every source.

Group by content tier

Pass ["tier"] to see how the volume breaks down by content tier:

"group_by": ["tier"]

"groups": {
  "tiers": [
    { "id": "premium-news", "documents": 412, "chunks": 889 },
    { "id": "podcasts", "documents": 156, "chunks": 198 },
    { "id": "regulatory-filings", "documents": 89, "chunks": 112 }
  ]
}

Content tiers are premium-news, corporate-communications, expert-interviews, earnings-transcripts, regulatory-filings, podcasts, web, and private-data. A document belongs to exactly one tier, so tier counts add up to the totals.

Group by package

Enterprise users on the unit-based consumption model can group by package to see which data packages contribute to the volume:

"group_by": ["package"]

"groups": {
  "packages": [
    { "id": "public", "documents": 672, "chunks": 889 },
    { "id": "podcasts", "documents": 156, "chunks": 198 }
  ]
}

The package id matches the ids shown in the Bigdata Store. A document can belong to several packages, so package counts may overlap and do not add up to the totals.

Prerequisites

• Python 3.9 or higher
• A Bigdata.com API key

System Dependencies

Matplotlib requires some system libraries for chart generation.

Ubuntu/Debian

sudo apt-get install python3-tk

macOS

# Usually pre-installed, but if needed:
brew install python-tk

Windows

Matplotlib should work out of the box with the pip installation.

Setup

1. Clone the repository

   git clone https://github.com/Bigdata-com/bigdata-docs-resources.git
   cd bigdata-docs-resources/how_to_guides/api_theme_volume
   
2. Create a virtual environment (recommended)

   python -m venv venv
   

   Activate the virtual environment:

   Linux/macOS

   source venv/bin/activate
   

   Windows

   venv\Scripts\activate
   
3. Install Python dependencies

   pip install -r requirements.txt
   
4. Configure API Key

   Copy the .env_template file to .env:

   cp .env_template .env
   

   Open the .env file and replace the placeholder value with your actual API key:

   BIGDATA_API_KEY=your_bigdata_api_key_here
   

   You can obtain your API key from the Bigdata Platform.

   Never commit your .env file to version control. Add it to your .gitignore file.

Usage

Run the script

Run the script with required date range parameters:

python theme_volume_evolution.py --start-date 2025-01-01 --end-date 2025-12-15

Command-Line Arguments

Argument	Short	Description	Required
--start-date	-s	Start date in format YYYY-MM-DD or YYYY-MM-DDTHH:MM:SSZ	Yes
--end-date	-e	End date in format YYYY-MM-DD or YYYY-MM-DDTHH:MM:SSZ	Yes
--theme	-t	Theme to search for (default: “Tariffs impact”)	No

Examples

Basic usage with default theme:

python theme_volume_evolution.py --start-date 2025-01-01 --end-date 2025-12-15

Using short flags:

python theme_volume_evolution.py -s 2025-01-01 -e 2025-12-15

With a custom theme:

python theme_volume_evolution.py -s 2025-01-01 -e 2025-12-15 --theme "Quantum computing chip"

View help:

python theme_volume_evolution.py --help

Output

The script generates a high-resolution (300 DPI) PNG chart with a timestamped filename:

{theme}_volume_evolution_YYYYMMDD_HHMMSS.png

The chart includes three subplots:

1. Unique Documents per Day: Daily document counts as bars with weekly average line
2. Chunks per Day: Daily chunk counts as bars with weekly average line
3. Sentiment per Day: Daily sentiment values as a line with weekly average line

Example Charts

Here are example charts generated for different themes:

Tariffs Impact

Theme: "Tariffs impact"

Quantum Computing

Theme: "Quantum computing chip"

Solid State Batteries

Theme: "Solid state batteries"

Greenland

Theme: "Greenland"

Venezuela

Theme: "Venezuela"