HeyGen

Build me a focused synthetic voice, dubbing and AI avatars workflow for the personal core of HeyGen. Requirements:

- Use Python 3.12, FastAPI, SQLite, ffmpeg, and a user-owned local TTS model. Model source audio, processing jobs, transcript or edit segments, and exported media; store source IDs and timestamps for each.
- Paid product context: Assemble a labeled avatar-style video from user-owned media without cloning real people. Build only this DIY scope: Turn user-authored text into a clearly labeled avatar-style video assembled from user-owned media with a local model, never cloning real people, and retain provenance for every output.
- Generate speech from text with voice, speed, pause, pronunciation, and segment controls.
- Create a timeline for audio, captions, uploaded visuals, and simple transitions.
- Store prompts, model identifiers, consent notes, and output hashes in a local provenance log.
- Use a local web page with input, progress, review, and export views. Required input or access: local TTS model; ffmpeg.
- Recovery: On invalid input or interrupted processing, keep the original record, show the failed step, and permit a safe retry.
- Acceptance: with one labelled sample, show the input, saved intermediate state, and exported result; verify this invariant: An invalid input or interrupted operation must retain the source and show a recoverable state; exported records must reload with the same IDs.
- Out of scope: proprietary avatars, lip sync, translation, rendering, templates, and rights operations; frontier voice or avatar model. Keep this a personal, inspectable workflow.
- Include a README with setup, a sample input, required keys or permissions, data location, and the supported scope.

# AGENTS.md — Build guide for HeyGen

## Project scope
Turn user-authored text into a clearly labeled avatar-style video assembled from user-owned media with a local model, never cloning real people, and retain provenance for every output.

Catalogue verdict: no. A consolation build is possible, but the paid product's decisive value sits outside a solo rebuild. For HeyGen, assemble a labeled avatar-style video from user-owned media without cloning real people. The hard boundary is proprietary avatars, lip sync, translation, rendering, templates, and rights operations, plus models, compute, rights, and safety operations.
Use the implementation prompt below to define the deliverable. Complete each phase's acceptance checks before extending the scope.

## Working agreement
- Inspect the repository and its existing instructions before choosing paths, dependencies or commands. Keep one coherent stack and explain changes to the proposed architecture.
- Plan a vertical slice that accepts a real input and produces the useful output described below. Persist only the state the prompt calls for; respect memory-only and upstream-managed workflows. Use fixtures only when they are clearly labelled.
- After scaffolding, document the actual install, development, check and build commands in README and keep them synchronized with the package or project manifest. Do not report commands as successful unless they ran.
- Work in small steps. At handoff, list implemented flows, checks actually performed, remaining blockers, and any credentials or provider setup the owner must supply.
- Do not publish, spend money, contact customers, delete source data or run irreversible migrations without the project owner's authorization.

## Prerequisites
- local TTS model
- ffmpeg
- GPU recommended
- voices the user has rights and consent to use

## Stack and architecture
- Use Python 3.12, FastAPI, SQLite, ffmpeg, and a user-owned local TTS model.
- Model source audio, processing jobs, transcript or edit segments, and exported media; store source IDs and timestamps for each.
- Interface for this synthetic voice, dubbing and AI avatars workflow: a local web page with input, progress, review, and export views.

## Security and data integrity
- On invalid input or interrupted processing, keep the original record, show the failed step, and permit a safe retry.
- Validate imported records and file paths; keep user data within the storage location named in the prompt.
- State the practical limit: proprietary avatars, lip sync, translation, rendering, templates, and rights operations.

## Agent implementation rules
- Project rule, data: Model source audio, processing jobs, transcript or edit segments, and exported media; store source IDs and timestamps for each.
- Project rule, behavior: Generate speech from text with voice, speed, pause, pronunciation, and segment controls.
- Project rule, recovery: On invalid input or interrupted processing, keep the original record, show the failed step, and permit a safe retry.

## Optional agent skills and references
No installable skill is listed for this guide; use the project rules and the stack documentation.

Read the linked SKILL.md and its dependencies before adding a skill. Select only the skills matching this project's runtime and task; their documentation does not supply API access, credentials or approval to perform external actions. Pin the reviewed revision where the tool supports it. Follow the chosen agent's documented project-level installation mechanism.

## Distribution ideas
These are optional planning notes. Obtain the owner's approval before publishing or contacting anyone.
- Demonstrate this synthetic voice, dubbing and AI avatars operation with sample or user-owned data: Generate speech from text with voice, speed, pause, pronunciation, and segment controls.
- Show a concrete result or export: Embed project metadata and a visible synthetic-media disclosure in exported assets.
- State the practical limit: proprietary avatars, lip sync, translation, rendering, templates, and rights operations.

## Engineering roadmap
1. Phase 1, architecture and data: Use Python 3.12, FastAPI, SQLite, ffmpeg, and a user-owned local TTS model. Model source audio, processing jobs, transcript or edit segments, and exported media; store source IDs and timestamps for each.
2. Phase 2, implement: Generate speech from text with voice, speed, pause, pronunciation, and segment controls.
3. Phase 3, implement: Create a timeline for audio, captions, uploaded visuals, and simple transitions.
4. Phase 4, review and output: Store prompts, model identifiers, consent notes, and output hashes in a local provenance log. Embed project metadata and a visible synthetic-media disclosure in exported assets.
5. Phase 5, recovery and acceptance: On invalid input or interrupted processing, keep the original record, show the failed step, and permit a safe retry. Verify this invariant with a saved fixture: An invalid input or interrupted operation must retain the source and show a recoverable state; exported records must reload with the same IDs. State the practical limit: proprietary avatars, lip sync, translation, rendering, templates, and rights operations.

## Paid-product capabilities outside this build
- proprietary avatars, lip sync, translation, rendering, templates, and rights operations
- frontier voice or avatar model
- licensed voice catalog
- real-time rendering fleet
- moderation, consent verification, and enterprise rights

## Implementation prompt
Build me a focused synthetic voice, dubbing and AI avatars workflow for the personal core of HeyGen. Requirements:

- Use Python 3.12, FastAPI, SQLite, ffmpeg, and a user-owned local TTS model. Model source audio, processing jobs, transcript or edit segments, and exported media; store source IDs and timestamps for each.
- Paid product context: Assemble a labeled avatar-style video from user-owned media without cloning real people. Build only this DIY scope: Turn user-authored text into a clearly labeled avatar-style video assembled from user-owned media with a local model, never cloning real people, and retain provenance for every output.
- Generate speech from text with voice, speed, pause, pronunciation, and segment controls.
- Create a timeline for audio, captions, uploaded visuals, and simple transitions.
- Store prompts, model identifiers, consent notes, and output hashes in a local provenance log.
- Use a local web page with input, progress, review, and export views. Required input or access: local TTS model; ffmpeg.
- Recovery: On invalid input or interrupted processing, keep the original record, show the failed step, and permit a safe retry.
- Acceptance: with one labelled sample, show the input, saved intermediate state, and exported result; verify this invariant: An invalid input or interrupted operation must retain the source and show a recoverable state; exported records must reload with the same IDs.
- Out of scope: proprietary avatars, lip sync, translation, rendering, templates, and rights operations; frontier voice or avatar model. Keep this a personal, inspectable workflow.
- Include a README with setup, a sample input, required keys or permissions, data location, and the supported scope.

## Completion evidence
Demonstrate the prompt's acceptance scenarios against the scoped workflow. Include setup from a clean checkout and failure recovery. Check persistence across restart and export/restore only for the state the prompt says to store; for memory-only tools, confirm that temporary content is discarded as specified. Record actual results and remaining limitations. A detailed plan alone does not establish a working replacement.