Skip to main content
It will only take you a few minutes. This guide walks you through: ✅ Set up authentication (API key)
✅ Create two sample files to upload
✅ Upload private files via the Content API
✅ Query bigdata.com with the Search API
Private & Secure: No LLM training on your data
Ready to get started? Run the examples below in your terminal (or open the notebook in Colab). All requests use the REST API with your API key in the X-API-KEY header. Open in Colab

​
Set up authentication

Send your API key in the X-API-KEY header on every request. See Authentication for details. Replace YOUR_API_KEY in the examples with your key.

​
Create two sample files to upload

Create the following two sample files in your local directory. File name data_science_research-2020-06.txt: 
RavenPack Data Science researchers recommend the following stocks in June 2020

Microsoft (NASDAQ: MSFT): Microsoft has been heavily investing in AI and cloud computing, which are key growth areas for the company.

Datadog (NASDAQ: DDOG): Datadog is a leading provider of monitoring and analytics solutions for cloud-based applications.

Oracle (NYSE: ORCL): Oracle is a major player in the enterprise software and cloud computing market.
File name soup_recipes-2020-06.txt: 
We recommend making chicken noodle soup with homemade chicken stock

​
Upload private files (Content API)

Upload the two files using the Content API: request a pre-signed URL and document id with POST /contents/v1/documents, then PUT the file to that URL. Bigdata enriches each document (extraction, structure and annotation of the content) and indexes it, making it available for Search and Research Agent. Use published_ts to set the document date (so we can narrow search to June 2020) and tags to filter by type later. 1. Request upload URL and upload the first file 
# Request a pre-signed URL (replace YOUR_API_KEY with your key)
RESP=$(curl -s -X POST 'https://api.bigdata.com/contents/v1/documents' \
  -H 'X-API-KEY: YOUR_API_KEY' \
  -d '{
    "file_name": "data_science_research-2020-06.txt",
    "published_ts": "2020-06-10T12:00:00Z",
    "tags": ["Data Science Research"],
    "share_with_org": false
  }')

# Extract the URL (e.g. with jq) and PUT the file
URL=$(echo $RESP | jq -r '.url')
curl -X PUT "$URL" --data-binary '@./data_science_research-2020-06.txt'
2. Request upload URL and upload the second file 
RESP=$(curl -s -X POST 'https://api.bigdata.com/contents/v1/documents' \
  -H 'X-API-KEY: YOUR_API_KEY' \
  -d '{
    "file_name": "soup_recipes-2020-06.txt",
    "published_ts": "2020-06-10T12:00:00Z",
    "tags": ["Cooking recipes"],
    "share_with_org": false
  }')

URL=$(echo $RESP | jq -r '.url')
curl -X PUT "$URL" --data-binary '@./soup_recipes-2020-06.txt'
After each PUT, Bigdata enriches and indexes the document. The POST response includes an id (content ID); use it with Get document metadata to poll status until it is completed. See Upload your own content for full details.

​
Query bigdata.com

Run a similarity search for “recommend stock” in June 2020. To search only your uploaded files, filter by category my_files. Use POST /v1/search with a query that has text (for similarity) and filters for timestamp and document category. To restrict results to your uploaded files, set category to my_files. Similarity search in June 2020 (all sources): 
curl -X POST 'https://api.bigdata.com/v1/search' \
  -H 'Content-Type: application/json' \
  -H 'X-API-KEY: YOUR_API_KEY' \
  --data '{
    "query": {
      "text": "recommend stock",
      "filters": {
        "timestamp": {
          "start": "2020-06-01T00:00:00.000Z",
          "end": "2020-06-30T23:59:59.999Z"
        }
      },
      "max_chunks": 10
    }
  }'
Search only your uploaded files (my_files): 
curl -X POST 'https://api.bigdata.com/v1/search' \
  -H 'Content-Type: application/json' \
  -H 'X-API-KEY: YOUR_API_KEY' \
  --data '{
    "query": {
      "text": "recommend stock",
      "filters": {
        "timestamp": {
          "start": "2020-06-01T00:00:00.000Z",
          "end": "2020-06-30T23:59:59.999Z"
        },
        "category": {
          "mode": "INCLUDE",
          "values": ["my_files"]
        }
      },
      "max_chunks": 10
    }
  }'
Narrow to the publication window of your files (2 seconds around 2020-06-10 12:00:00): 
curl -X POST 'https://api.bigdata.com/v1/search' \
  -H 'Content-Type: application/json' \
  -H 'X-API-KEY: YOUR_API_KEY' \
  --data '{
    "query": {
      "text": "recommend stock",
      "filters": {
        "timestamp": {
          "start": "2020-06-10T11:59:59.000Z",
          "end": "2020-06-10T12:00:01.000Z"
        },
        "category": {
          "mode": "INCLUDE",
          "values": ["my_files"]
        }
      },
      "max_chunks": 10
    }
  }'
You should see your two documents (e.g. data_science_research-2020-06.txt and soup_recipes-2020-06.txt) in the results. Filter by tag (e.g. only “Data Science Research”): Use the tag filter with any_of to restrict to documents that have a specific tag. See Search API: tag filter. 
curl -X POST 'https://api.bigdata.com/v1/search' \
  -H 'Content-Type: application/json' \
  -H 'X-API-KEY: YOUR_API_KEY' \
  --data '{
    "query": {
      "text": "recommend stock",
      "filters": {
        "timestamp": {
          "start": "2020-06-01T00:00:00.000Z",
          "end": "2020-06-30T23:59:59.999Z"
        },
        "category": {
          "mode": "INCLUDE",
          "values": ["my_files"]
        },
        "tag": {
          "any_of": ["Data Science Research"]
        }
      },
      "max_chunks": 10
    }
  }'

​
Summary

Congratulations! You have uploaded private files with the Content API and queried them via the Search API.
Related documentation
Click on the playgrounds below, use the source selector and filter by My Files to test and retrieve your uploaded content.

Search Service Playground

Search across your private content and other sources. In the playground, open the source selector and choose My Files to limit results to your uploaded documents.

Research Agent Playground

Run research over your private content and real-time data. In the playground, use the source selector and filter by My Files to ground answers in your documents.