Multimodal Fusion for Speaker Diarization
Combine visual features (face detection, lip movement analysis) with audio features to improve speaker diarization accuracy in video files. Use OpenCV for face detection and lip movement tracking, then fuse visual cues with audio-based speaker embeddings. Essential when processing video files with multiple visible speakers or when audio-only diarization needs visual validation.
Install
npx skills add UCSB-NLP-Chang/Skill-Usage --skill Multimodal Fusion for Speaker Diarizationgit clone --depth 1 https://github.com/UCSB-NLP-Chang/Skill-Usage.git /tmp/Skill-Usage
mkdir -p ~/.claude/skills
cp -r /tmp/Skill-Usage/tasks/speaker-diarization-subtitles/environment/skills/multimodal-fusion ~/.claude/skills/Multimodal-Fusion-for-Speaker-DiarizationPersonal skills live in ~/.claude/skills. Use .claude/skills in a project folder to share a skill with your team.
From the SKILL.md
# Multimodal Fusion for Speaker Diarization
## Overview
When working with video files, you can significantly improve speaker diarization by combining audio features with visual features like face detection and lip movement analysis.
## When to Use
- Processing video files (not just audio)
- Multiple speakers visible on screen
- Need to disambiguate speakers with similar voices
- Improve accuracy by leveraging visual cues
## Visual Feature Extraction
### Face Detection
```python
import cv2
import numpy as np
# Initialize face detector
face_cascade = cv2.CascadeClassifier(
cv2.data.haarcascades + 'haarcascade_frontalface_default.xml'
)
# Process video frames
cap = cv2.VideoCapture(video_path)
fps = cap.get(cv2.CAP_PROP_FPS)
faces_by_time = {}
frame_count = 0
frame_skip = max(1, int(fps / 2)) # Process every other frame
while cap.isOpened():
ret, frame = cap.read()
if not ret:
break
if frame_count % frame_skip == 0:
timestamp = frame_count / fps
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
faces = face_cascade.detectMultiScale(gray, 1.1, 4)
faces_by_time[timestamp] = len(faces)
frame_count += 1
cap.release()
```
### Lip Movement DetectionDetails
- Skill name
Multimodal Fusion for Speaker Diarization- Repository
- UCSB-NLP-Chang/Skill-Usage
- Path
tasks/speaker-diarization-subtitles/environment/skills/multimodal-fusion/SKILL.md- Repo updated
- 8 April 2026
FAQ
What does the Multimodal Fusion for Speaker Diarization skill do?
Combine visual features (face detection, lip movement analysis) with audio features to improve speaker diarization accuracy in video files. Use OpenCV for face detection and lip movement tracking, then fuse visual cues with audio-based speaker embeddings. Essential when processing video files with multiple visible speakers or when audio-only diarization needs visual validation.
How do I install the Multimodal Fusion for Speaker Diarization skill?
skills CLI (Claude Code, Cursor, Codex and more): npx skills add UCSB-NLP-Chang/Skill-Usage --skill Multimodal Fusion for Speaker Diarization. Skills go in ~/.claude/skills (personal) or .claude/skills (project), and Claude loads them automatically when a task matches the description.
Is Multimodal Fusion for Speaker Diarization popular?
It lives in UCSB-NLP-Chang/Skill-Usage, which has 49 GitHub stars and 5 forks.
Related skills
Pubchem Database
49UCSB-NLP-Chang/Skill-Usage
Query PubChem via PUG-REST API/PubChemPy (110M+ compounds). Search by name/CID/SMILES, retrieve properties, similarity/substructure searches, bioactivity, for cheminformatics.
Pymatgen
49UCSB-NLP-Chang/Skill-Usage
Materials science toolkit. Crystal structures (CIF, POSCAR), phase diagrams, band structure, DOS, Materials Project integration, format conversion, for computational materials science.
Rdkit
49UCSB-NLP-Chang/Skill-Usage
Cheminformatics toolkit for fine-grained molecular control. SMILES/SDF parsing, descriptors (MW, LogP, TPSA), fingerprints, substructure search, 2D/3D generation, similarity, reactions. For standard workflows with simpler interface, use datamol (wrapper around RDKit). Use rdkit for advanced control, custom sanitization, specialized algorithms.
Obj Exporter
49UCSB-NLP-Chang/Skill-Usage
Three.js OBJExporter utility for exporting 3D geometry to Wavefront OBJ format. Use when converting Three.js scenes, meshes, or geometries to OBJ files for use in other 3D software like Blender, Maya, or MeshLab.
Mesh Analysis
49UCSB-NLP-Chang/Skill-Usage
Analyzes 3D mesh files (STL) to calculate geometric properties (volume, components) and extract attribute data. Use this skill to process noisy 3D scan data and filter debris.
Object Counter
49UCSB-NLP-Chang/Skill-Usage
Count occurrences of an object in the image using computer vision algorithm.
Browse the most popular Claude skills, all images, video & audio skills, or write your own with the skills generator.