Scraping YouTube comments for sentiment analysis
Scrape YouTube comments and you build the foundation for one of the most expressive sentiment-analysis datasets on the public web. YouTube comments are unstructured, multilingual, opinion-rich, and reflect real reactions to media events, product launches, public figures, and content trends. The scraping landscape is shaped by three things: an official YouTube Data API that offers generous quota for most use cases, a comment system with sophisticated bot-detection that makes browser-based scraping non-trivial, and a sentiment-analysis layer that benefits dramatically from modern transformer models compared to bag-of-words approaches.
This guide covers practical patterns for building a YouTube comment-and-sentiment dataset that supports research projects from brand monitoring to political discourse analysis.
YouTube Data API as the primary path
The YouTube Data API v3 is the canonical access path for YouTube comments. The API exposes endpoints for video search, video detail, comment threads, and comment replies. Authentication uses OAuth 2.0 or API keys. The default quota is 10,000 units per day per project; comment-thread reads cost 1 unit each. For most research projects, that means roughly 10,000 comment threads per day per project.
from googleapiclient.discovery import build
def get_comments(video_id: str, api_key: str, max_results: int = 100):
youtube = build("youtube", "v3", developerKey=api_key)
request = youtube.commentThreads().list(
part="snippet,replies",
videoId=video_id,
maxResults=max_results,
textFormat="plainText",
)
all_comments = []
while request:
response = request.execute()
for item in response.get("items", []):
top = item["snippet"]["topLevelComment"]["snippet"]
all_comments.append({
"comment_id": item["id"],
"author": top["authorDisplayName"],
"text": top["textOriginal"],
"like_count": top.get("likeCount", 0),
"published_at": top["publishedAt"],
"reply_count": item["snippet"].get("totalReplyCount", 0),
})
request = youtube.commentThreads().list_next(request, response)
return all_comments
For research on a specific video, the API path is sufficient. For research at scale across millions of videos, the quota becomes a binding constraint. Quota expansion through Google’s standard process is realistic for legitimate research projects, but the approval process takes 4-8 weeks.
Browser-based fallback for high-volume needs
For volumes beyond the YouTube Data API quota, browser-based scraping fills the gap. YouTube’s comment system loads comments through a continuation-token-based pagination scheme that you can replicate with HTTP calls. The implementation is non-trivial because the continuation tokens are encrypted and YouTube rotates the encryption schema periodically.
The yt-dlp project (an actively maintained fork of youtube-dl) handles the comment-fetching protocol and is the practical baseline for browser-based scraping. yt-dlp’s API extracts comments via a Python interface that wraps the browser-side endpoints.
import yt_dlp
def fetch_comments_ytdlp(video_url: str, max_comments: int = 1000):
ydl_opts = {
"skip_download": True,
"writeinfojson": False,
"getcomments": True,
"quiet": True,
"extractor_args": {"youtube": {"max_comments": [str(max_comments)]}},
}
with yt_dlp.YoutubeDL(ydl_opts) as ydl:
info = ydl.extract_info(video_url, download=False)
return info.get("comments", [])
For sustained scraping with yt-dlp, residential proxies and request-rate limiting are needed because YouTube enforces bot detection on the comment endpoints. Datacenter IPs work for short bursts but get throttled within hours.
Sentiment analysis approaches
Once you have comment text, the sentiment analysis layer is where the analytical value materializes. Three approaches with different cost and accuracy profiles:
Lexicon-based approaches (VADER, AFINN) score sentiment using word-level dictionaries. They run at millions of comments per second on a single core and require no model hosting. Accuracy is moderate (around 65-75% on labeled benchmarks) and they handle short text well. They struggle with sarcasm, negation, and code-switching.
Classical ML approaches (logistic regression on TF-IDF, fastText) produce higher accuracy (75-85%) and can be trained on domain-specific data. They are still cheap to run but require training data and ongoing model maintenance.
Transformer-based approaches (RoBERTa, DistilBERT, multilingual XLM-R) produce the highest accuracy (85-92% on English benchmarks). They are more expensive per inference but with batching can still run at thousands of comments per second on a single GPU. For most research projects, fine-tuning a small transformer on your specific domain is the right tradeoff.
from transformers import pipeline
sentiment_pipeline = pipeline("sentiment-analysis", model="cardiffnlp/twitter-roberta-base-sentiment-latest")
def score_sentiment(comment_text: str) -> dict:
result = sentiment_pipeline(comment_text)[0]
return {"label": result["label"], "score": result["score"]}
For multilingual research, models like XLM-RoBERTa or the more recent multilingual sentiment models from Cardiff NLP handle 30+ languages with reasonable accuracy. For specialized domains (gaming, politics, beauty), domain-specific fine-tuning on a few thousand labeled comments produces meaningful accuracy gains.
Schema for comment-and-sentiment snapshots
CREATE TABLE youtube_comment (
comment_id VARCHAR(64) PRIMARY KEY,
video_id VARCHAR(32) NOT NULL,
author_channel_id VARCHAR(64),
text TEXT,
like_count INT,
reply_count INT,
published_at TIMESTAMP,
sentiment_label VARCHAR(16),
sentiment_score DECIMAL(5,4),
language VARCHAR(8)
);
CREATE INDEX comment_video_idx ON youtube_comment(video_id);
For longitudinal sentiment analysis, snapshot the like_count periodically because comments accumulate engagement over time. A high-engagement comment from launch day on a music video is qualitatively different from a low-engagement comment, and the engagement count is itself a useful weighting signal in sentiment aggregations.
For broader pattern guidance, see our residential proxy provider ranking and our Python scraping libraries ranking. The Stanford NLP toolkit is also a useful reference for the underlying linguistic models.
Detecting and routing around bot challenges
When YouTube flag your traffic, the response is usually a Cloudflare or vendor interrogation page rather than a clean HTTP error. Your scraper needs to detect this content-type swap explicitly. Look for the signature cf-mitigated header, the presence of __cf_chl_ cookies, or HTML containing Just a moment....
def is_challenged(response) -> bool:
if response.status_code in (403, 503):
return True
if "cf-mitigated" in response.headers:
return True
if "__cf_chl_" in response.headers.get("set-cookie", ""):
return True
body = response.text[:2000].lower()
return "just a moment" in body or "checking your browser" in body
When you detect a challenge, do not retry on the same IP for at least 30 minutes. Mark that IP as cooling and route subsequent requests to a different IP in your pool. Aggressive retries on a flagged IP cause the cooling window to extend. For pages that absolutely must be fetched, have a fallback path that uses a headless browser. Most production setups maintain a 95/5 split between the lightweight HTTP path and the browser fallback path.
Operational monitoring and alerting
Every production scraper needs three monitoring layers. The first is per-IP success rate over a 5-minute window, alerting if any IP drops below 80%. The second is parser error rate, alerting if more than 1% of fetched pages fail to extract the canonical fields. The third is data freshness, alerting if your downstream consumers see snapshots more than 24 hours old.
import time
from collections import deque
class IPHealthTracker:
def __init__(self, window_seconds: int = 300):
self.window = window_seconds
self.events = {}
def record(self, ip: str, success: bool):
bucket = self.events.setdefault(ip, deque())
now = time.time()
bucket.append((now, success))
while bucket and bucket[0][0] < now - self.window:
bucket.popleft()
def success_rate(self, ip: str) -> float:
bucket = self.events.get(ip)
if not bucket:
return 1.0
return sum(1 for _, ok in bucket if ok) / len(bucket)
Wire this into Prometheus or your existing observability stack so the on-call engineer sees IP degradation as it happens rather than after the daily snapshot fails. For long-running operations, IP rotation triggered by the health tracker is more reliable than fixed rotation schedules.
Pipeline orchestration and scheduling
For any non-trivial YouTube comments scraping operation, a dedicated orchestration layer is the difference between a script you babysit and a service that runs unattended. The two strong open-source choices in 2026 are Prefect 3 and Dagster.
from prefect import flow, task
@task(retries=3, retry_delay_seconds=60)
def fetch_source(source_id: str, page: int):
return crawl_one_page(source_id, page)
@flow(name="youtube-comments-daily-sweep")
def daily_sweep(source_ids: list):
futures = []
for sid in source_ids:
for page in range(1, 30):
futures.append(fetch_source.submit(sid, page))
return [f.result() for f in futures]
Run the flow on a cadence aligned to how dynamic the underlying data is. For YouTube comments where records change intraday, a 4-6 hour cadence catches meaningful movements. For longer-cycle data, daily is sufficient.
Data quality monitoring patterns
Beyond per-IP success rate, every snapshot should pass a small battery of data quality checks before being considered authoritative. Structural checks verify that every required field is present and of the expected type. Distributional checks compare the current snapshot against recent history. Semantic checks compare related fields for consistency.
def quality_check(snapshot: list[dict]) -> list[str]:
errors = []
if not snapshot:
errors.append("empty snapshot")
return errors
avg_yesterday = get_yesterday_avg_size()
if len(snapshot) < avg_yesterday * 0.7:
errors.append("snapshot size below threshold")
return errors
Run quality checks as a separate flow that gates promotion of the snapshot from staging to production. A snapshot that fails quality checks should be quarantined for human review.
Cost optimization strategies
Proxy bandwidth is usually the dominant cost in a production scraping operation. Three optimization patterns consistently reduce cost without hurting data quality. The first is request deduplication: serve cached responses when consumers ask for the same record within the same hour. The second is conditional GET using ETag or If-Modified-Since headers when supported. The third is selective field hydration when the upstream API supports field selection.
For workloads above 100 GB of monthly proxy bandwidth, these three optimizations together reduce cost by 40-60% without changing the analytical output. The engineering effort is modest and the payback period is usually under a month at production volume.
End-to-end pipeline architecture
A production-grade scraping pipeline has four layers: collection, parsing, storage, and serving. The collection layer handles the network conversation and knows nothing about data shape. The parsing layer transforms raw bytes into structured records and owns the schema. The storage layer holds the canonical snapshots in a query-optimized format like DuckDB or ClickHouse. The serving layer exposes the data to consumers and should be denormalized and pre-aggregated where possible. Decoupling these layers also enables independent scaling.
Legal and compliance considerations
Public YouTube comments data is generally treated as fair to scrape in most jurisdictions, but always confine your collection to non-personal data. For commercial deployment, document your basis for processing, your data retention period, and your purpose limitation. The W3C Web Annotation guidance and similar published frameworks remain useful starting points for documenting your approach.
Sample analytics queries
-- Volume trend over the last 30 days
SELECT date_trunc('day', snapshot_at) AS day, COUNT(*) AS records
FROM snapshot
WHERE snapshot_at > now() - interval '30 days'
GROUP BY 1 ORDER BY 1;
-- Source distribution
SELECT source, COUNT(*) AS records
FROM snapshot
WHERE snapshot_at > now() - interval '7 days'
GROUP BY source
ORDER BY records DESC;
Add a category share view, a source concentration view, and a price-volatility view (where applicable) and you have a solid foundation for a YouTube comments intelligence product.
Versioning your scraper for source evolution
Every YouTube comments source evolves its schema regularly. Stamp every snapshot row with the scraper version that produced it. Downstream analytics can filter by version when they need consistent semantics across a time range. Pair this with a small registry table that documents what each scraper version did differently so debugging unexpected metric jumps becomes tractable.
Caching strategy and incremental crawls
Full daily snapshots scale linearly with source size, which becomes expensive at multi-million record scale. Most production deployments shift from full snapshots to incremental refreshes after the initial ramp using freshness deadline, volatility, and business priority signals. Priority-driven scheduling reduces total request volume by 60-80% compared to blind full snapshots.
Building a brand sentiment tracker
The most common analytical product on top of YouTube comment scraping is a brand sentiment tracker that aggregates sentiment across all comments mentioning a brand across the YouTube ecosystem. The tracker requires three components: a search layer that finds comments mentioning the brand, a sentiment scoring layer, and an aggregation layer that produces daily and weekly sentiment indices.
def brand_sentiment_index(brand_keywords, comments_df):
matched = comments_df[comments_df['text'].str.lower().str.contains('|'.join(brand_keywords), na=False)]
return matched.groupby('snapshot_date').agg(
positive_count=('sentiment_label', lambda x: (x == 'positive').sum()),
negative_count=('sentiment_label', lambda x: (x == 'negative').sum()),
net_sentiment=('sentiment_score', 'mean'),
)
For commercial brand intelligence, the headline metrics are net sentiment trend (week-over-week movement), volume trend (mention count week-over-week), and the comment-on-comment-of-comment ratio (a useful proxy for controversy intensity).
Topic modeling for emergent themes
Beyond sentiment, topic modeling reveals what people are actually talking about. Modern approaches use sentence embeddings clustered via HDBSCAN or BERTopic to surface coherent topics from comment corpora.
from sentence_transformers import SentenceTransformer
from bertopic import BERTopic
model = BERTopic(embedding_model=SentenceTransformer("all-MiniLM-L6-v2"))
topics, probs = model.fit_transform(comments_list)
For brand monitoring use cases, topic modeling reveals emergent issues that don’t yet trigger sentiment shifts. A topic cluster around “service quality” or “pricing changes” is an early indicator of brand health risk, often before the sentiment metric moves measurably.
Code-switching and multilingual comment handling
YouTube comments are heavily multilingual and frequently code-switch within a single comment (English plus Spanish, English plus Hindi, etc). Sentiment models trained on monolingual data perform poorly on code-switched text. The practical solution is to use multilingual transformer models (XLM-R, multilingual BERT) that handle the language-mixing natively. For research focused on a specific market, fine-tuning a multilingual base model on local code-switched data produces the strongest accuracy.
Author behavior and bot detection
YouTube comment sections include a meaningful fraction of bot or coordinated inauthentic activity, especially around politically sensitive content or during product launches. Filtering inauthentic comments before sentiment aggregation produces more credible analytical outputs.
The standard heuristics are: comment author with very few total comments and zero subscribers (likely bot), comment posted within seconds of video publication on a video with millions of views (likely staged), and identical comment text across multiple videos (likely coordinated). Each heuristic has false positives but combined they catch most coordinated activity.
Working with hosted scraping services
For projects where the engineering investment of running a self-hosted scraping pipeline is not justified, hosted scraping services like ScrapingBee, ZenRows, ScrapeOps, and Apify offer a different cost-and-control tradeoff. These services maintain proxy pools and headless browser fleets and expose a per-request API that abstracts away the infrastructure.
The cost model is per-request rather than per-byte. For low-volume projects (under 100,000 requests per month), the hosted services are typically cheaper than rolling your own proxy and browser infrastructure. For high-volume projects, the math flips because the per-request markup adds up at scale.
import httpx
async def scrape_via_hosted(target_url: str, api_key: str):
proxy_url = f"https://api.scrapingbee.com/api/v1/?api_key={api_key}&url={target_url}&render_js=true"
async with httpx.AsyncClient(timeout=60) as c:
r = await c.get(proxy_url)
return r.text
For research projects with bounded scope, the hosted-service path is often the fastest way to ship. For ongoing production pipelines, the self-hosted path tends to win on per-request cost and on long-term flexibility.
Long-term archival and data retention
Snapshot data accumulates rapidly. A daily snapshot of even a moderate-sized dataset produces gigabytes per month and terabytes per year. The storage layer needs a clear lifecycle policy. Hot data (last 90 days) sits in your primary store for fast queries. Warm data (90 days to 2 years) sits in a cheaper columnar archive (Parquet on S3, BigQuery, ClickHouse cold storage). Cold data (older than 2 years) sits in compressed archive form, accessed rarely.
def lifecycle_archival(snapshot_age_days):
if snapshot_age_days <= 90:
return "hot"
elif snapshot_age_days <= 730:
return "warm"
else:
return "cold"
The lifecycle policy interacts with your data retention obligations. Some jurisdictions impose maximum retention periods on certain data types. Document the retention policy in writing and audit compliance quarterly.
Common pitfalls when scraping YouTube comments
Three issues catch most teams. The first is ordering instability. YouTube’s comment thread default order is ‘Top comments’ which is engagement-weighted and changes minute by minute. Switch to ‘Newest first’ (order=time in the Data API) for any longitudinal analysis or your snapshots will show the same comment at different positions and your ‘new comment rate’ calculation will be unreliable.
The second is reply-thread truncation. The Data API returns top-level comments with up to five replies inline. Threads with more than five replies require a separate commentThreads.list call per parent. A scraper that ignores deep threads undercounts engagement on viral videos by 30-60%.
The third is sentiment classifier drift across languages. A model fine-tuned on English YouTube comments performs 15-30% worse on Spanish, Portuguese, or Japanese comments. For multilingual datasets, use a per-language classifier or a multilingual model (XLM-R, mBERT) and validate accuracy per-language with a labeled sample before reporting aggregate sentiment.
FAQ
Is scraping YouTube comments legal?
The YouTube Data API is officially supported and its terms of service allow analytical use. Browser-based scraping outside the API violates YouTube terms of service; YouTube enforces these terms through technical countermeasures rather than legal action for non-commercial research. Confine your collection to non-personal data and document your basis for processing.
How do I handle comment author personal data?
Comment authors have public display names that they chose for that comment system. The display name is technically personal data but the privacy expectation is low. Hash the author identifier in your dataset rather than storing the raw display name where possible.
Can I scrape live chat from YouTube live streams?
Live chat uses a separate API surface and accumulates rapidly. Real-time analysis is possible but requires stream-processing infrastructure. For most research projects, the chat replay (available after the stream ends as a static archive) is the practical access path.
What about YouTube Shorts comments?
Shorts comments use the same API surface as regular YouTube comments. The content shape is different (very short, more emoji-heavy, more reaction-style) which affects sentiment model performance. Domain-specific fine-tuning helps.
How fresh is the data through the API?
The YouTube Data API reflects current state with a few-second cache lifetime. New comments appear within seconds. For real-time sentiment monitoring, polling at 1-5 minute intervals is the typical pattern.
Does the YouTube Data API quota allow production sentiment monitoring?
The 10,000-unit daily quota covers roughly 10,000 comment-list calls. For brand monitoring across hundreds of videos, request a higher quota via Google’s quota-extension form.
How do I detect bot or coordinated-inauthentic comment activity?
Cluster comments by author + posting cadence + text similarity. Genuine viewers post at human cadence with diverse phrasing; bot rings post in bursts with templated phrasing.
To build broader social media intelligence pipelines, browse the ai-data-collection category for tooling reviews and framework deep dives.
last updated: May 17, 2026