L

AI Video Lip Sync Tools

LatentSync: Open-Source Lip-Sync Software Review

github.com · checked 21 Oct 2026

Self-hosted Linux software that retimes a video's lips to follow new audio.

How it plugs in

—Public API
YesOpen source
YesFree tier

Signal on AI Video Lip Sync Tools7.3#2 of 28

ByteDance developed LatentSync as an open-source method for synchronizing a video's mouth movements with another audio track. Whisper supplies audio embeddings to latent diffusion models built on Stable Diffusion. Inference is available through a local Gradio interface or the command line, while the documented setup targets Linux and a capable GPU. Output is listed at 720p or below.

LatentSync home page

Review

In a line

LatentSync is an open-source system for audio-driven lip synchronization. Audio directly guides generated mouth movements, without an intermediate motion representation.

What it does

Whisper turns mel spectrograms into embeddings, which enter the U-Net through cross-attention as the diffusion process creates corresponding lip motion. Stable Diffusion handles the audio-visual relationship directly.

For version 1.6, training used 512 × 512 videos to address soft-looking teeth and lips. The stated output ceiling is 720p.

Increasing inference steps can improve appearance but extends generation time. A higher guidance scale may improve alignment while introducing distortion or jitter.

How it connects

This Linux-oriented, self-hosted project offers an API, a local Gradio interface and command-line inference. The release supplies inference code, checkpoints, data-processing tools and training code.

Its documented pipeline uses PySceneDetect to detect scenes and InsightFace landmarks to transform faces. Setup creates a Conda environment, installs FFmpeg and downloads Whisper plus LatentSync checkpoints from Hugging Face.

Hosted options are also available through third-party services, including an API endpoint that accepts source video and reference audio. Replicate exposes LatentSync through its API.

Good fit

LatentSync is suited to users seeking local lip-sync processing, editable processing or training code, and Gradio or command-line access.

The repository supplies preprocessing, inference, U-Net-training and file-list scripts. Its efficient second-stage training configuration is described for consumer-grade GPUs such as an RTX 3090.

You can also compare it with other AI video lip-sync tools when choosing a hosted or local approach.

Watch for

Hardware demands are substantial. LatentSync 1.5 lists 8 GB of VRAM for inference, while 1.6 lists 18 GB; training configurations range from 20 GB to 55 GB depending on stage and resolution.

The efficient 20 GB Stage 2 configuration may sacrifice a little visual quality and temporal consistency relative to the 30 GB setup. An initial installation can take over an hour and is Linux-focused.

The repository identifies Apache License 2.0, while the Hugging Face model card identifies openrail++. Check both licenses for your intended use. The Hugging Face page says version 1.6 is not deployed by an Inference Provider.

Pricing

LatentSync is available at no cost as an open-source release. The package contains inference code and checkpoints, a data-processing pipeline and training code.

Verdict

Version 1.5 uses less memory for inference than version 1.6.

Consider another option if you need a turnkey cloud service, lighter hardware requirements or resolution beyond the documented 720p limit.

Our take

LatentSync is a capable option for technical users who want free, open-source lip syncing on their own Linux machine. It offers local and command-line inference plus the code needed for custom data processing and training. The trade-off is substantial GPU memory: version 1.6 needs at least 18 GB for inference, and training can require up to 55 GB. Installation also involves a Conda environment and several pinned dependencies. Choose it for control and local processing; look elsewhere for a simpler, hosted workflow.

  • No-cost open-source release with inference, data and training components
  • Inference works through a local Gradio interface or the command line
  • Version 1.6's training targets sharper teeth and lip detail
  • Preprocessing and training workflows can be customized
  • Version 1.6 lists 18 GB of VRAM as its inference minimum
  • Training setups span 20–55 GB of VRAM
  • Linux installation brings Conda, FFmpeg and pinned dependencies
  • Additional inference steps take longer; strong guidance can create jitter
  • Pick it ifSelf-hosted lip-sync work on Linux GPUs; Users who need editable preprocessing or training code; Teams comfortable managing local model checkpoints
  • Skip it ifYou need a simple hosted service with no GPU setup; Your hardware has less than 8 GB of VRAM; You require output above the documented 720p limit

Pricing

The open-source release is free.

  • Open-source releaseInference code and checkpoints · data processing pipeline · training codeFree

As github.com lists them, checked 8 Oct 2026.

Questions

Is LatentSync free?

Yes. The open-source package includes inference code, checkpoints, a data pipeline and training code.

What hardware does inference need?

The stated minimum is 8 GB of VRAM for LatentSync 1.5 and 18 GB for version 1.6.

Can I run it without a graphical app?

Yes. The project includes both a local Gradio application and a command-line inference script.

What operating system does it use?

The documented setup targets Linux. It creates a Conda environment, installs FFmpeg and requires the libgl1 package.

Can I train LatentSync?

Yes. Training and preprocessing code are included, with documented configurations needing 20–55 GB of VRAM.

What output quality or size is listed?

The documented output is 720p or lower. Version 1.6 training used 512 × 512 videos to address blurry teeth and lips.

Does a hosted API exist?

Third-party options exist. Layer documents an authenticated endpoint for source video and reference audio, and Replicate exposes LatentSync through its API.

What licenses apply?

The source repository uses Apache License 2.0. The Hugging Face model card labels the model openrail++.