Multimodal Fusion for Speaker Diarization

Combine visual features (face detection, lip movement analysis) with audio features to improve speaker diarization accuracy in video files. Use OpenCV for face detection and lip movement tracking, then fuse visual cues with audio-based speaker embeddings. Essential when processing video files with multiple visible speakers or when audio-only diarization needs visual validation.

49 stars 5 forksImages, Video & Audio

Install

skills CLI (Claude Code, Cursor, Codex and more)
npx skills add UCSB-NLP-Chang/Skill-Usage --skill Multimodal Fusion for Speaker Diarization
Manual install
git clone --depth 1 https://github.com/UCSB-NLP-Chang/Skill-Usage.git /tmp/Skill-Usage
mkdir -p ~/.claude/skills
cp -r /tmp/Skill-Usage/tasks/speaker-diarization-subtitles/environment/skills/multimodal-fusion ~/.claude/skills/Multimodal-Fusion-for-Speaker-Diarization

Personal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.

From the SKILL.md

# Multimodal Fusion for Speaker Diarization

## Overview

When working with video files, you can significantly improve speaker diarization by combining audio features with visual features like face detection and lip movement analysis.

## When to Use

- Processing video files (not just audio)
- Multiple speakers visible on screen
- Need to disambiguate speakers with similar voices
- Improve accuracy by leveraging visual cues

## Visual Feature Extraction

### Face Detection

```python
import cv2
import numpy as np

# Initialize face detector
face_cascade = cv2.CascadeClassifier(
    cv2.data.haarcascades + 'haarcascade_frontalface_default.xml'
)

# Process video frames
cap = cv2.VideoCapture(video_path)
fps = cap.get(cv2.CAP_PROP_FPS)
faces_by_time = {}

frame_count = 0
frame_skip = max(1, int(fps / 2))  # Process every other frame

while cap.isOpened():
    ret, frame = cap.read()
    if not ret:
        break

    if frame_count % frame_skip == 0:
        timestamp = frame_count / fps
        gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
        faces = face_cascade.detectMultiScale(gray, 1.1, 4)
        faces_by_time[timestamp] = len(faces)

    frame_count += 1

cap.release()
```

### Lip Movement Detection
Read the full skill on GitHub

Details

Skill name
Multimodal Fusion for Speaker Diarization
Path
tasks/speaker-diarization-subtitles/environment/skills/multimodal-fusion/SKILL.md
Repo updated
8 April 2026

FAQ

What does the Multimodal Fusion for Speaker Diarization skill do?

Combine visual features (face detection, lip movement analysis) with audio features to improve speaker diarization accuracy in video files. Use OpenCV for face detection and lip movement tracking, then fuse visual cues with audio-based speaker embeddings. Essential when processing video files with multiple visible speakers or when audio-only diarization needs visual validation.

How do I install the Multimodal Fusion for Speaker Diarization skill?

skills CLI (Claude Code, Cursor, Codex and more): npx skills add UCSB-NLP-Chang/Skill-Usage --skill Multimodal Fusion for Speaker Diarization. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.

Is Multimodal Fusion for Speaker Diarization popular?

It lives in UCSB-NLP-Chang/Skill-Usage, which has 49 GitHub stars and 5 forks.

Related skills

Pubchem Database

49

UCSB-NLP-Chang/Skill-Usage

Query PubChem via PUG-REST API/PubChemPy (110M+ compounds). Search by name/CID/SMILES, retrieve properties, similarity/substructure searches, bioactivity, for cheminformatics.

Databases

Pymatgen

49

UCSB-NLP-Chang/Skill-Usage

Materials science toolkit. Crystal structures (CIF, POSCAR), phase diagrams, band structure, DOS, Materials Project integration, format conversion, for computational materials science.

Research & Science

Rdkit

49

UCSB-NLP-Chang/Skill-Usage

Cheminformatics toolkit for fine-grained molecular control. SMILES/SDF parsing, descriptors (MW, LogP, TPSA), fingerprints, substructure search, 2D/3D generation, similarity, reactions. For standard workflows with simpler interface, use datamol (wrapper around RDKit). Use rdkit for advanced control, custom sanitization, specialized algorithms.

Frontend & Design

Obj Exporter

49

UCSB-NLP-Chang/Skill-Usage

Three.js OBJExporter utility for exporting 3D geometry to Wavefront OBJ format. Use when converting Three.js scenes, meshes, or geometries to OBJ files for use in other 3D software like Blender, Maya, or MeshLab.

Images, Video & Audio

Mesh Analysis

49

UCSB-NLP-Chang/Skill-Usage

Analyzes 3D mesh files (STL) to calculate geometric properties (volume, components) and extract attribute data. Use this skill to process noisy 3D scan data and filter debris.

Images, Video & Audio

Object Counter

49

UCSB-NLP-Chang/Skill-Usage

Count occurrences of an object in the image using computer vision algorithm.

Images, Video & Audio

Browse the most popular Claude skills, all images, video & audio skills, or write your own with the skills generator.