Overview
Key Features
-
Multi-Tenant Workspace & Team Collaboration:
Role-based access control (Owner vs. Member) allowing team collaboration on pitch projects, session management, secure member invitations, and ownership transfers. -
Direct-to-Storage Resilient Ingestion:
Offloads heavy video and slide deck uploads from the backend by issuing time-limited presigned URLs for direct transfer to S3 / Cloudflare R2 object storage. Client-side Web Workers compute streaming SHA-256 chunk checksums before upload to ensure end-to-end payload integrity. -
Real-Time Telemetry & Progress Streaming:
Live pipeline execution updates delivered over Server-Sent Events (SSE), featuring monotonic event sequencing and automatic stream recovery via Last-Event-ID to withstand transient network drops. -
Multimodal Speaker Diarization & Active Speaker Attribution:
Combines acoustic diarization with computer vision lip-motion tracking (Lip-MAR) to isolate individual presenters (SPEAKER_00, SPEAKER_01) and provides an interactive interface for team members to map themselves to their respective presentation segments. -
Context-Grounded Follow-Up Q&A:
Synthesizes pitch transcripts with presentation decks and supporting documents to generate challenging, context-grounded questions. An interactive browser-based voice recorder captures team answers with automated transcription and quality analysis. -
Evidence-Linked Rubric Scoring & Exportable Reports:
Scores pitch sessions across standard competition rubrics (delivery, problem-solution fit, technical depth, and Q&A responsiveness), anchoring every critique to specific video timestamps and slide numbers. Generates downloadable PDF reports for offline team review.
User Roles
-
Team Owner:
Full administrative control over the team workspace. Can create and delete projects, manage team members, transfer ownership, initiate pitch evaluations, and execute privacy-preserving data erasure requests. -
Team Member:
Can view workspace projects, upload practice recordings and decks, review telemetry, participate in interactive voice Q&A sessions, and access final evaluation reports.
System Workflow
-
Workspace & Asset Preparation:
Team members select or create a project workspace and select the target pitch recording (MP4/WebM) along with presentation slides (PDF). -
Direct Signed Upload:
The browser calculates a SHA-256 chunk checksum in a background Web Worker and requests presigned PUT URLs from the backend, uploading media directly to object storage without placing transit load on the API server. -
Multimodal Analysis:
The backend initiates an asynchronous AI pipeline that extracts audio tracks, executes speech-to-text transcription, runs audio diarization, tracks presenter lip movement via computer vision, and parses slide content. -
Interactive Speaker Mapping:
The team reviews identified speaker clusters alongside audio and visual previews, attributing each detected presenter segment to verified team members. -
Grounded Follow-Up Q&A:
Using vector embeddings and Retrieval-Augmented Generation (RAG), the system generates grounded judge questions. Presenters record their answers directly in the browser. -
Evidence-Linked Scoring & Report Generation:
The evaluation engine scores both the pitch delivery and Q&A responses against structured competition rubrics, compiling an evidence-backed report with timestamped video moments and slide references.
Technologies Used
-
Frontend
- Next.js (App Router): Modern React server components, dynamic routing, and API rewrites.
- React & TypeScript: Interactive, accessible UI built with strict static typing.
- TailwindCSS: Clean, responsive glassmorphic interface design system.
- TanStack React Query: Efficient server-state synchronization, caching, and optimistic mutations.
- Server-Sent Events (SSE): Low-latency unidirectional event streaming for real-time inference tracking.
- Web Workers: Multi-threaded client-side SHA-256 chunk hashing for pre-upload validation.
-
Backend
- FastAPI (Python): High-throughput asynchronous REST API control plane.
- Clean Hexagonal Architecture: Strict four-layer design (Domain, Application, API, Infrastructure) separating domain invariants from framework dependencies.
- SQLAlchemy (Async): Fully asynchronous database operations and Unit-of-Work patterns.
- Pydantic: Fast data validation, schema enforcement, and OpenAPI generation.
-
AI/ML & Media Processing
- Speech-to-Text & Diarization: Audio extraction, transcription, and speaker clustering.
- Computer Vision (Lip-MAR): Lip-motion active speaker detection for video-audio alignment.
- Vector RAG: Document parsing, semantic chunking, and grounded question generation.
- FFmpeg & FFprobe: Media stream verification, audio waveform extraction, and format sanitization.
-
Database & Storage
- PostgreSQL & pgvector: Relational data persistence, transactional integrity, and vector similarity search.
- Redis: Asynchronous AI job queue and low-latency event cache.
- Cloudflare R2 / AWS S3 / MinIO: Object storage for pitch media, slides, answer recordings, and generated reports.
Challenges and Learnings
-
Direct-to-Cloud Ingestion with Integrity Guarantees:
Eliminated backend bandwidth bottlenecks by routing large video uploads directly to S3/R2 presigned URLs. Solved payload tampering risks by computing SHA-256 digests in browser Web Workers and enforcing strict storage-level checksum verification upon completion. -
Resilient Real-Time Event Streaming:
Addressed connection interruptions during multi-minute AI inference pipelines by designing an SSE streaming protocol with monotonic event sequence IDs. On reconnect, the client provides Last-Event-ID, allowing the backend to replay missed events from Redis cache without loss of state. -
Multimodal Presenter Attribution:
Bridged acoustic diarization with computer vision lip-motion analysis to disambiguate overlapping speakers and assign presentation intervals accurately to individual team members. -
Optimistic Concurrency Control (OCC):
Safeguarded concurrent multi-user team workflows by implementing HTTP ETag and If-Match preconditions on mutable resources, preventing lost updates when multiple team members collaborate simultaneously. -
Domain-Driven Architectural Isolation:
Enforced strict boundaries between the domain layer, use-case application services, HTTP routes, and infrastructure adapters, ensuring the core business logic remains independent of external frameworks and third-party services.