Algo SEO Crawl

Implement a web crawler pipeline covering URL discovery, fetching, parsing, and storage. Use this skill when the user needs to build a site crawler, audit website structure, or collect web data systematically — even if they say 'scrape a website', 'crawl all pages', or 'site audit spider'.

248 stars 30 forksData & Analytics

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add asgard-ai-platform/skills --skill algo-seo-crawl
Manual install
git clone --depth 1 https://github.com/asgard-ai-platform/skills.git /tmp/skills
mkdir -p ~/.claude/skills
cp -r /tmp/skills/algo-seo-crawl ~/.claude/skills/algo-seo-crawl

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Web Crawler

## Overview

A web crawler systematically traverses web pages by discovering URLs, fetching content, parsing HTML, and storing results. Uses BFS or priority-based frontier management. Performance is I/O-bound, typically limited by politeness constraints rather than compute.

## When to Use

**Trigger conditions:**
- Building a site audit tool to discover all pages and their link structure
- Collecting structured data from websites at scale
- Mapping site architecture for SEO analysis

**When NOT to use:**
- When you need data from a single API endpoint (use HTTP client directly)
- When a sitemap.xml provides all needed URLs (parse sitemap instead)

## Algorithm

```
IRON LAW: Respect robots.txt and Rate Limits
A crawler MUST:
1. Parse and obey robots.txt before crawling any path
2. Enforce crawl-delay (default 1s if unspecified)
3. Identify itself with a descriptive User-Agent
Ignoring these is unethical and will get your IP blocked.
```

### Phase 1: Input Validation
Parse seed URLs, fetch and parse robots.txt for each domain, set crawl scope (same-domain, subdomain, or cross-domain).
**Gate:** Valid seed URLs, robots.txt rules loaded, scope defined.

### Phase 2: Core Algorithm
1. Initialize URL frontier with seed URLs (priority queue or FIFO)
2. Dequeue URL, check: not visited, allowed by robots.txt, within scope
3. Fetch page with timeout and retry logic, respect crawl-delay
4. Parse HTML: extract links (normalize, deduplicate), extract content/metadata
Read the full skill on GitHub

Details

Skill name
algo-seo-crawl
Path
algo-seo-crawl/SKILL.md
License
MIT
Repo updated
6 June 2026

FAQ

What does the Algo SEO Crawl skill do?

Implement a web crawler pipeline covering URL discovery, fetching, parsing, and storage. Use this skill when the user needs to build a site crawler, audit website structure, or collect web data systematically — even if they say 'scrape a website', 'crawl all pages', or 'site audit spider'.

How do I install the Algo SEO Crawl skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add asgard-ai-platform/skills --skill algo-seo-crawl. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Algo SEO Crawl popular?

It lives in asgard-ai-platform/skills, which has 248 GitHub stars and 30 forks.

Related skills

Algo Ad Bidding

248

asgard-ai-platform/skills

Implement and select ad bidding strategies from manual CPC to automated target-CPA and target-ROAS. Use this skill when the user needs to choose a bidding strategy, set up automated bidding, or optimize bid parameters — even if they say 'what bidding strategy should I use', 'target CPA setup', or 'smart bidding configuration'.

Marketing, Sales & Business30

Algo Ad Budget

248

asgard-ai-platform/skills

Optimize advertising budget allocation across campaigns using marginal returns analysis. Use this skill when the user needs to distribute budget across multiple campaigns, optimize spend pacing, or maximize overall ROAS under budget constraints — even if they say 'how to split my ad budget', 'campaign budget optimization', or 'diminishing returns on ad spend'.

Marketing, Sales & Business30

Algo Ad Ctr

248

asgard-ai-platform/skills

Build CTR prediction models for estimating ad click-through rates from features. Use this skill when the user needs to predict click probability, build an ad ranking model, or evaluate ad creative performance — even if they say 'predict click rate', 'ad relevance scoring', or 'which ad will get more clicks'.

Marketing, Sales & Business30

Cs Analytics

248

asgard-ai-platform/skills

Measure and optimize customer service performance using CSAT, NPS, CES, First Contact Resolution, and text mining on support tickets. Use this skill when the user needs to evaluate CS team performance, identify top complaint drivers, optimize staffing, or build CS dashboards — even if they say 'is our CS team doing well', 'what are customers complaining about', 'how many agents do we need', or 'build a CS dashboard'.

Data & Analytics30

Algo Mfg Spc

248

asgard-ai-platform/skills

Implement Statistical Process Control charts to monitor production process stability. Use this skill when the user needs to detect process shifts, set control limits, or distinguish common cause from special cause variation — even if they say 'process monitoring', 'control chart', or 'is our process in control'.

Data & Analytics30

Data Dashboard Design

248

asgard-ai-platform/skills

Design effective data dashboards with proper KPI hierarchy, chart type selection, and interactive features. Use this skill when the user needs to create a dashboard, choose the right visualizations, organize metrics for different audiences, or evaluate dashboard tools — even if they say 'build a dashboard', 'our reports are confusing', 'which chart should I use', or 'executives can't find the metrics they need'.

Data & Analytics30

Browse the most popular Claude skills, all data & analytics skills, or write your own with the skills generator.