Spark Python Data Source
Build custom Python data sources for Apache Spark using the PySpark DataSource API — batch and streaming readers/writers for external systems. Use this skill whenever someone wants to connect Spark to an external system (database, API, message queue, custom protocol), build a Spark connector or plugin in Python, implement a DataSourceReader or DataSourceWriter, pull data from or push data to a system via Spark, or work with the PySpark DataSource API in any way. Even if they just say "read from X in Spark" or "write DataFrame to Y" and there's no native connector, this skill applies.
Install
/plugin marketplace add databricks/databricks-agent-skillsThe repo is a Claude Code plugin marketplace. Run /plugin afterwards to pick the plugin that includes this skill.
npx skills add databricks/databricks-agent-skills --skill spark-python-data-sourcegit clone --depth 1 https://github.com/databricks/databricks-agent-skills.git /tmp/databricks-agent-skills
mkdir -p ~/.claude/skills
cp -r /tmp/databricks-agent-skills/experimental/spark-python-data-source ~/.claude/skills/spark-python-data-sourcePersonal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.
From the SKILL.md
# spark-python-data-source Build custom Python data sources for Apache Spark 4.0+ to read from and write to external systems in batch and streaming modes. ## Instructions You are an experienced Spark developer building custom Python data sources using the PySpark DataSource API. Follow these principles and patterns. ### Core Architecture Each data source follows a flat, single-level inheritance structure: 1. **DataSource class** — entry point that returns readers/writers 2. **Base Reader/Writer classes** — shared logic for options and data processing 3. **Batch classes** — inherit from base + `DataSourceReader`/`DataSourceWriter` 4. **Stream classes** — inherit from base + `DataSourceStreamReader`/`DataSourceStreamWriter` See [implementation-template.md](references/implementation-template.md) for the full annotated skeleton covering all four modes (batch read/write, stream read/write). ### Spark-Specific Design Constraints These are specific to the PySpark DataSource API and its driver/executor architecture — general Python best practices (clean code, minimal dependencies, no premature abstraction) still apply but aren't repeated here.
Details
- Skill name
spark-python-data-source- Repository
- databricks/databricks-agent-skills
- Path
experimental/spark-python-data-source/SKILL.md- Repo updated
- 18 November 2026
FAQ
What does the Spark Python Data Source skill do?
Build custom Python data sources for Apache Spark using the PySpark DataSource API — batch and streaming readers/writers for external systems. Use this skill whenever someone wants to connect Spark to an external system (database, API, message queue, custom protocol), build a Spark connector or plugin in Python, implement a DataSourceReader or DataSourceWriter, pull data from or push data to a system via Spark, or work with the PySpark DataSource API in any way. Even if they just say "read from X in Spark" or "write DataFrame to Y" and there's no native connector, this skill applies.
How do I install the Spark Python Data Source skill?
Claude Code plugin: /plugin marketplace add databricks/databricks-agent-skills. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.
Is Spark Python Data Source popular?
It lives in databricks/databricks-agent-skills, which has 407 GitHub stars and 103 forks.
Related skills
Databricks AI Functions
407databricks/databricks-agent-skills
Use Databricks built-in AI Functions (ai_classify, ai_decide, ai_extract, ai_summarize, ai_mask, ai_translate, ai_fix_grammar, ai_gen, ai_analyze_sentiment, ai_similarity, ai_parse_document, ai_prep_search, ai_query, ai_forecast) to add AI capabilities directly to SQL and PySpark pipelines without managing model endpoints. Also covers document parsing and building custom RAG pipelines (parse → prep_search → index → query).
Databricks AI Runtime
407databricks/databricks-agent-skills
Databricks AI Runtime, the `databricks air` CLI commands for submitting and managing GPU training workloads on Databricks serverless compute. Use for: writing and submitting `databricks air` workload YAML, choosing an accelerator type and Python environment, reading Unity Catalog tables as training data with UCDataset, passing hyperparameters and secrets, configuring retries, checking run status, listing/cancelling runs, streaming a run's logs and watching its progress, debugging failed runs, GPU pools, multi-container workloads, custom Docker image setup, and converting a workload to a Databr
Databricks Aibi Dashboards
407databricks/databricks-agent-skills
Create Databricks AI/BI dashboards. Must use when creating, updating, or deploying Lakeview dashboards as Databricks Dashboard have a unique json structure. CRITICAL: You MUST test ALL SQL queries via CLI BEFORE deploying. Follow guidelines strictly.
What Happened
240entireio/skills
Explain why code looks the way it does by tracing the latest change for a file range or pasted snippet through `git blame` and deduplicated `entire checkpoint explain` lookups. Use when the user asks what happened, says "tell me why" about a code block, is confused about a section of code, asks "wtf is going on", "why is this like this", "why was this changed", or wants provenance for a specific file block.
Releasing Clickup CLI
132krodak/clickup-cli
Publishes a new version of clickup-cli to npm, updates Homebrew tap, writes release notes, and syncs the agent skill. Use when releasing a new version, bumping version, or verifying a release.
Antfu
398JetBrains/skills
Anthony Fu's opinionated tooling and conventions for JavaScript/TypeScript projects. Use when setting up new projects, configuring ESLint/Prettier alternatives, monorepos, library publishing, or when the user mentions Anthony Fu's preferences.
Browse the most popular Claude skills, all software development skills, or write your own with the skills generator.