Narrative Miners

Uncover the Stories That Drive Markets

Introduction

This notebook demonstrates how advanced narrative mining reveals evolving market stories across multiple document types. We will track the “AI Bubble Concerns” narrative as it emerges and evolves across news, earnings calls, and regulatory filings – highlighting the difference between public discourse and corporate communications.

The bigdata-research-tools package provides a specialised class, NarrativeMiner, for narrative mining. We can specify the document_type parameter to be DocumentType.NEWS, DocumentType.TRANSCRIPTS, DocumentType.FILINGS to be able to

• Analyze web-based news content
• Examine earnings call and event transcripts
• Explore SEC Filings from EDGAR

Each Narrative Miner instance follows the same workflow:

1. Define narrative labels which encompass a theme
2. Retrieve content using Bigdata’s search capabilities
3. Label content with LLMs to identify narrative matches
4. Analyze the results to reveal patterns and insights

Ready to get started? Let’s dive in!

Setup and Imports

Below is the Python code required for setting up our environment and importing necessary libraries.

from IPython.display import display, HTML, IFrame
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
from datetime import datetime
import os
from scipy.ndimage import gaussian_filter1d
import plotly
import plotly.graph_objects as go
import warnings
from IPython.display import Image, display
import plotly.io as pio
pio.renderers.default = 'notebook'
plotly.offline.init_notebook_mode()

from bigdata_client import Bigdata
from bigdata_client.daterange import RollingDateRange
from bigdata_research_tools.workflows import NarrativeMiner
from bigdata_client.models.sources import Source
from bigdata_client.models.search import DocumentType

Extract and Print Key Insights

Extract key insights from the narrative mining data and display them.

Extract insights

# Key Insights from Narrative Mining
def extract_narrative_insights():
    """
    Extract key insights from our narrative mining data.
    """
    news_score = calculate_source_scores(news_df)
    transcript_score = calculate_source_scores(transcripts_df)
    filing_score = calculate_source_scores(filings_df)

    peak_news_month = news_score.idxmax().strftime('%B %Y')
    peak_transcript_month = transcript_score.idxmax().strftime('%B %Y')
    peak_filing_month = filing_score.idxmax().strftime('%B %Y')

    news_narrative_counts = news_df['Label'].value_counts()
    transcript_narrative_counts = transcripts_df['Label'].value_counts()
    filing_narrative_counts = filings_df['Label'].value_counts()

    top_news_narrative = news_narrative_counts.index[0]
    top_transcript_narrative = transcript_narrative_counts.index[0]
    top_filing_narrative = filing_narrative_counts.index[0]

    total_news_mentions = len(news_df)
    total_transcript_mentions = len(transcripts_df)
    total_filing_mentions = len(filings_df)

    news_peaks = pd.Series(news_score).nlargest(3)
    transcript_peaks = pd.Series(transcript_score).nlargest(3)
    filing_peaks = pd.Series(filing_score).nlargest(3)

    avg_lag_days = []
    for news_date in news_peaks.index:
        closest_filing_date = min(filing_peaks.index, key=lambda x: abs((x - news_date).days))
        lag_days = (closest_filing_date - news_date).days
        avg_lag_days.append(lag_days)
    avg_lag = np.mean(avg_lag_days)

    return {
        "peak_news_month": peak_news_month,
        "peak_transcript_month": peak_transcript_month,
        "peak_filing_month": peak_filing_month,
        "top_news_narrative": top_news_narrative,
        "top_transcript_narrative": top_transcript_narrative,
        "top_filing_narrative": top_filing_narrative,
        "total_news_mentions": total_news_mentions,
        "total_transcript_mentions": total_transcript_mentions,
        "total_filing_mentions": total_filing_mentions,
        "avg_lag_days": int(avg_lag)
    }

insights = extract_narrative_insights()

print("## AI Bubble Narrative Key Insights\n")
print(f"Peak month for news coverage: {insights['peak_news_month']}")
print(f"Peak month for earnings call mentions: {insights['peak_transcript_month']}")
print(f"Peak month for regulatory filing mentions: {insights['peak_filing_month']}")
print(f"\nDominant narrative in news: \"{insights['top_news_narrative']}\"")
print(f"Dominant narrative in earnings calls: \"{insights['top_transcript_narrative']}\"")
print(f"Dominant narrative in regulatory filings: \"{insights['top_filing_narrative']}\"")
print(f"\nTotal narrative mentions in news: {insights['total_news_mentions']}")
print(f"Total mentions in earnings calls: {insights['total_transcript_mentions']}")
print(f"Total mentions in regulatory filings: {insights['total_filing_mentions']}")
print(f"\nAverage lag between peaks in news coverage peaks and SEC filings: {insights['avg_lag_days']} days")

## AI Bubble Narrative Key Insights

Peak month for news coverage: November 2023
Peak month for earnings call mentions: December 2023
Peak month for regulatory filing mentions: August 2024

Dominant narrative in news: "Analysts are questioning the timeline for AI-driven profits"
Dominant narrative in earnings calls: "Tech CEOs acknowledge AI implementation challenges amid high expectations"
Dominant narrative in regulatory filings: "Current AI investments may not generate predicted financial returns"

Total narrative mentions in news: 64
Total mentions in earnings calls: 641
Total mentions in regulatory filings: 744

Average lag between peaks in news coverage peaks and SEC filings: 280 days

Conclusion

The Narrative Miners reveal important patterns in how the AI bubble narrative evolved across information sources:

Timing and Intensity Variations:

• News media (red line) shows a major spike in AI bubble concerns around October 2023, reaching the highest peak on the chart, followed by mostly below-average coverage until a smaller resurgence in late 2024.
• Earnings calls (teal line) show cyclical attention to bubble concerns, with executives addressing the topic most prominently in early 2024, followed by diminishing attention through 2025.
• SEC filings (purple line) demonstrate the most volatile pattern, with multiple significant spikes in early/mid-2024 and again in late 2024/early 2025, suggesting ongoing regulatory concern.

Narrative Progression:

• Media coverage led the initial bubble narrative in late 2023, potentially triggering the corporate response visible in early 2024 earnings calls.
• Corporate executives’ discussions peaked early but diminished over time, while SEC filing mentions increased in frequency and intensity throughout 2024-2025.
• By early 2025, SEC filings show the strongest ongoing concerns about an AI bubble, while both media coverage and earnings call mentions have declined significantly.

Enjoy exploring and extending your narrative analysis!