io.github.RayAKaan/ocular-audio-mcp

OcularAudio MCP

Universal local-first media intelligence for AI agents.

OcularAudio lets MCP-compatible AI systems watch, listen to, search, inspect, compare, and reason over media. It turns URLs, streams, and local media into reusable timestamped evidence: metadata, transcripts, frames, OCR, retrieval results, multimodal moments, and cross-video intelligence.

v1.3.0 · MIT

Product

What it does

• Universal media intake: web URLs, direct media, local files, live/stream sources, and a broad set of social, video, audio and archive platforms.
• Speech understanding: timestamped captions with local Whisper fallback when captions are unavailable.
• Visual understanding: intelligent screenshots, exact frames, frame bursts, crops, visual indexing and perceptual similarity.
• On-screen text: optional Tesseract OCR produces searchable visual text evidence.
• Evidence search: timestamp-aware lexical, semantic and hybrid retrieval over stable, provenance-preserving canonical evidence records.
• Multimodal analysis: aligns transcript, visual and OCR evidence around the same moments.
• Batch intelligence: process multiple sources, search across them, and generate cross-video comparisons with failure isolation.
• Agentic execution: turn analysis requests into bounded plans with retries, timeouts, audit records and health reporting.
• Capability modes: Lite, Intelligence, Deep and Auto control processing depth and response richness.
• Local caching: reuse processed evidence and manage cache state through CLI or MCP.
• Operational awareness: capabilities, health, readiness, security state, metrics and the full protocol contract are machine-readable.
• MCP interoperability: tools, resources and prompts over stdio or Streamable HTTP.
• Production deployment: bearer-token protection, host/origin controls, Docker deployment, persistent cache and readiness checks.

Best use cases

Research and lecture analysis · product demonstrations · competitor analysis · interviews · technical talks · UI walkthroughs · multi-video comparison · evidence-grounded AI investigation.

Capability matrix

Canonical evidence architecture

All source artifacts are normalized into one local manifest per source. Evidence records have stable content-derived IDs, modality labels, timestamp bounds, content hashes, provenance pointers, and metadata. Transcript search, timelines, moment inspection, and hybrid retrieval share the same canonical IDs; OCR and visual-frame records are indexed alongside transcript evidence. Manifests are written atomically, validated on read, deduplicated by ID, and refreshed from the existing transcript and visual caches without requiring a migration or a database. Clearing a source removes its transcript, canonical manifest, visual index, and frame artifacts together. Manifest metadata intentionally stores a bounded summary rather than the source URL or credentials.

Four-level intelligence engine

Every request is resolved through one deterministic policy engine. Auto resolves to Lite, Intelligence, or Deep based on query intent, requested features, source count, media duration, and an optional time budget. The decision is returned as machine-readable metadata with explicit limits, disabled features, capability gaps, and adjustments.

Modes enforce evidence caps, frame limits, source limits, retry counts, concurrency, sampling intervals, and runtime ceilings. Missing OpenCV, FFmpeg, or Tesseract is reported as a capability gap; optional unavailable features are disabled rather than represented as successful. Explicit source-limit violations fail with a clear error instead of silently dropping sources. Source coverage remains shared across modes.

The CLI's existing --mode flag now reaches the Python runtime. Use --budget-seconds to impose a caller-specified runtime ceiling (0.001-900 seconds). MCP planning, agentic execution, hybrid search, visual indexing, multimodal analysis, and batch analysis expose the same decision contract. Use --mode deep for comprehensive evidence or --mode lite for a fast overview.

Quick start

NPM / MCP

Release status: this repository contains source version 1.3.0, but that version is not yet published to npm. The latest published package is 1.2.1, so the npx examples below install 1.2.1 until a separate 1.3.0 release is authorized and published. To test the current checkout, clone the repository, install the Python dependencies from requirements.txt, then run node bin/cli.js from the checkout.

npx -y ocular-audio-mcp

Local MCP hosts can launch the same command. Example:

{ "mcpServers": { "ocular-audio": { "command": "npx", "args": ["-y", "ocular-audio-mcp"] } } }

Claude Code:

claude mcp add ocular-audio -- npx -y ocular-audio-mcp

Codex CLI:

codex mcp add ocular-audio -- npx -y ocular-audio-mcp

Direct CLI

npx ocular-audio "https://www.youtube.com/watch?v=dQw4w9WgXcQ"; npx ocular-audio --check npx ocular-audio --json "URL" npx ocular-audio --visual-index --ocr "URL" npx ocular-audio --multimodal-search "pricing" "URL" npx ocular-audio --plan "Find the pricing discussion and inspect its visual evidence" npx ocular-audio --agentic "Find every pricing discussion and inspect the relevant slides"

Technical overview

MCP host / agent | MCP tools + prompts + resources | Universal source resolver | Evidence: metadata + transcript + timestamps + chapters / | \ visual OCR retrieval \ | / multimodal moments | batch intelligence | agentic execution | operations / audit / readiness

Core modules

• media_resolver.py + universal_sources.py: source classification, platform families, streams, local media and analysis policy.
• evidence_index.py: timestamped evidence segmentation, chapter association and deterministic retrieval.
• evidence_store.py: canonical transcript/chapter/metadata/visual/OCR records, stable evidence IDs, provenance, integrity-checked manifests, atomic persistence and lifecycle cleanup.
• visual_index.py: frame sampling, bursts, crops, perceptual hashes, descriptors and OCR evidence.
• semantic_index.py: local TF-IDF/lexical/semantic and hybrid retrieval.
• multimodal_index.py: transcript/visual/OCR timestamp alignment and ranking.
• batch_index.py: bounded multi-source execution, persistence, search and comparison.
• agentic_index.py: deterministic planning, retries, timeouts, audit records and execution health.
• capability_modes.py: Lite, Intelligence, Deep and Auto policy resolution.
• protocol_contract.py: machine-readable MCP interoperability contract.
• deployment_security.py: controlled HTTP authentication and transport security.
• runtime_ops.py: bounded metrics, latency observations, readiness and audit retention.

MCP surface

33 tools

Media/evidence: get_ocular_audio_transcript, get_ocular_audio_video_context, search_ocular_audio_video, get_ocular_audio_video_timeline, inspect_ocular_audio_moment, query_ocular_audio_evidence, get_ocular_audio_metadata, get_ocular_audio_chapters, inspect_ocular_audio_source, get_ocular_audio_capabilities.

Visual/OCR: get_ocular_audio_video_screenshots, index_ocular_audio_video_visuals, search_ocular_audio_visuals, get_ocular_audio_video_frame, get_ocular_audio_video_frame_burst, crop_ocular_audio_video_frame.

Retrieval/multimodal: search_ocular_audio_hybrid, analyze_ocular_audio_multimodal, inspect_ocular_audio_multimodal_moment, search_ocular_audio_cache.

Batch: analyze_ocular_audio_batch, search_ocular_audio_videos, compare_ocular_audio_videos, get_ocular_audio_batch, list_ocular_audio_batches.

Agentic/operations: plan_ocular_audio_analysis, run_ocular_audio_analysis, get_ocular_audio_health, get_ocular_audio_audit, get_ocular_audio_contract, get_ocular_audio_identity.

Cache: list_ocular_audio_cache, clear_ocular_audio_cache.

7 resources

• ocularaudio://capabilities
• ocularaudio://modes
• ocularaudio://health
• ocularaudio://contract
• ocularaudio://security
• ocularaudio://readiness
• ocularaudio://metrics

4 prompts

• inspect_video
• search_video_evidence
• review_visual_evidence
• compare_videos

Deployment

Local stdio

stdio is the default local MCP transport and requires no HTTP authentication.

Streamable HTTP

npx ocular-audio --transport streamable-http --host 127.0.0.1 --port 8000

The MCP endpoint is /mcp. For network exposure, configure authentication, real issuer/resource URLs, allowed hosts and allowed origins. Do not expose an unauthenticated instance to an untrusted network.

Docker

cp .env.example .env docker compose -f docker/docker-compose.yml up --build -d

The supplied deployment uses Python 3.11, FFmpeg, a persistent cache volume, a read-only container filesystem, dropped capabilities, no-new-privileges, restart policy and a healthcheck.

For internet-facing deployments, use TLS and an organization identity provider. The built-in static bearer verifier is intended for controlled deployments and can be replaced by JWT validation or RFC 7662 introspection.

Runtime requirements

• Python 3.11+ recommended for the release environment.
• Node.js 18+ for the NPM CLI.
• FFmpeg for media processing.
• OpenCV for visual processing.
• faster-whisper for local transcription fallback.
• Tesseract is optional for OCR.
• Network access is required for remote sources.

The NPM package does not hide native Python/system prerequisites; OcularAudio is a local media-processing server rather than a hosted API.

Cookies and restricted media

Operator-supplied cookies.txt can be used for sources that require an authenticated browser session. Never commit cookies, access tokens or credentials.

Release v1.3.0

npm install npm run check npm pack

This creates ocular-audio-mcp-1.3.0.tgz. Inspect it before publishing:

npm publish ocular-audio-mcp-1.3.0.tgz --access public

See RELEASE_v1.3.0.md. A manual GitHub Actions workflow can also build and upload the exact tarball as an artifact.

Validation

python -m unittest discover -s tests -v node --check bin/cli.js npm run check

CI additionally validates the MCP SDK surface, tool/resource/prompt registration, Streamable HTTP authorization behavior, compilation, CLI syntax and package metadata.

Design principles

• Local-first: no mandatory cloud inference API.
• Evidence-first: preserve timestamps and source evidence.
• Composable: each analysis layer builds on the previous evidence layer.
• Bounded: steps, retries, concurrency, metrics cardinality and audit retention have explicit limits.
• MCP-native: tools, resources and prompts are protocol primitives.
• Deterministic infrastructure: OcularAudio provides execution/evidence primitives while the host remains responsible for model-level reasoning.

License

MIT. See LICENSE.