Twitter X Scraping

Use when scraping public Twitter/X timelines or lists through Nitter-style mirrors with axios and Playwright fallback, including anti-bot challenge handling and pagination limits.

201 stars 12 forksData & Analytics

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add swyxio/skills --skill twitter-x-scraping
Manual install
git clone --depth 1 https://github.com/swyxio/skills.git /tmp/skills
mkdir -p ~/.claude/skills
cp -r /tmp/skills/twitter-x-scraping ~/.claude/skills/twitter-x-scraping

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Skill: Scraping Nitter.net for Tweets

This codebase scrapes [nitter.net](https://nitter.net) — an open-source Twitter frontend — to collect tweets from a list or profile within a time window. The technique works without a Twitter API key.

## Core Methodology

### 1. URL Structure

Nitter mirrors Twitter's URL layout:

| Target | Nitter path |
|--------|------------|
| Twitter List | `/i/lists/<listId>` |
| User Profile | `/<handle>` |
| Paginated | append `?cursor=<cursor>` |

### 2. Fetch Strategy: axios → Playwright fallback

Start with plain HTTP (`axios`). If nitter serves an anti-bot challenge page (detected by the string `"Verifying your request"` in the HTML), permanently switch to a headed Playwright/Chrome session for the rest of the run.

```ts
// challengeSolver.ts — detect challenge
const CHALLENGE_MARKER = 'Verifying your request';
export function isChallengePage(html: string): boolean {
  return html.includes(CHALLENGE_MARKER);
}

// scrapeNitterList.ts — switch modes on detection
if (isChallengePage(html)) {
  useBrowserMode = true;
  const browserHtml = await fetchWithBrowser(url, debug);
  return { html: browserHtml, url };
}
```

### 3. Browser Anti-bot Bypass

When challenge is detected, launch a real system Chrome (not headless) with automation signals suppressed:
Read the full skill on GitHub

Details

Skill name
twitter-x-scraping
Repository
swyxio/skills
Path
twitter-x-scraping/SKILL.md
License
MIT
Repo updated
16 November 2026

FAQ

What does the Twitter X Scraping skill do?

Use when scraping public Twitter/X timelines or lists through Nitter-style mirrors with axios and Playwright fallback, including anti-bot challenge handling and pagination limits.

How do I install the Twitter X Scraping skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add swyxio/skills --skill twitter-x-scraping. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Twitter X Scraping popular?

It lives in swyxio/skills, which has 201 GitHub stars and 12 forks.

Related skills

Accelevents API

201

swyxio/skills

Use when reading or updating AI Engineer Europe speaker records through the Accelevents REST API, especially for full-record PUT updates, auth header quirks, and preserving existing speaker fields.

Software Development12

Accelevents Speaker Sync

201

swyxio/skills

Use when website speaker, session, schedule, room, track, or headshot changes must be synchronized back to Accelevents for AI Engineer Europe 2026.

AI, LLMs & Agents12

Agents SDK

201

swyxio/skills

Build or debug stateful applications that specifically use Cloudflare's `agents` package, including Agent classes, routing, callable methods, synchronized state, scheduling, AIChatAgent, durable execution, or Agents SDK client hooks. Do not trigger for generic AI-agent design, the OpenAI Agents SDK, or ordinary Durable Object work that does not use the Cloudflare Agents SDK.

AI, LLMs & Agents12

Web Scraper

150

guia-matthieu/clawfu-skills

Extract structured data from websites. Use when: collecting competitor pricing; scraping product listings; extracting contact information; gathering research data; monitoring website changes

Data & AnalyticsIn 1 list32

Statistical Distance

191

jwynia/agent-skills

Transform clichéd story elements by pushing along the emotional vector toward statistical edges. Use when first instincts are too predictable, when elements feel generic, or when you need the core methodology for avoiding statistical-center defaults.

Data & Analytics24

Data Visualization

223

NeverSight/learn-skills.dev

Data visualization with chart selection, color theory, and annotation best practices. Covers chart types (bar, line, scatter, heatmap), axes rules, and storytelling with data. Use for: charts, graphs, dashboards, reports, presentations, infographics, data stories. Triggers: data visualization, chart, graph, data chart, bar chart, line chart, scatter plot, data viz, visualization, dashboard chart, infographic data, data presentation, chart design, plot, heatmap, pie chart alternative

Data & Analytics42

Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.