Video Converter
Microservices video-to-audio pipeline with queues and Kubernetes
In plain English
Upload a video, get the audio back as an MP3. The interesting part is that the work is split across independent services so heavy conversions never freeze the app.
My role
Solo architect and developer
Timeline
3 months
Team
Solo
Status
Open source
Section 01
The problem
Converting video is slow and CPU-hungry. If you do it inside the web request, one upload blocks the server, the user stares at a spinner, and a browser timeout loses the whole job. Scaling that naive design means scaling the entire application just to get more conversion capacity.
Section 02
How I approached it
I split the system into small services that talk through a message queue instead of calling each other directly. The gateway accepts the upload and immediately returns. A message goes onto RabbitMQ. A dedicated converter service picks it up whenever it has capacity, does the work, and publishes a completion message that triggers a notification.
Outcome
Uploads return instantly, conversion capacity scales independently of the web tier, and a crashed worker means a retried message rather than a lost file.
Section 03
Going deeper
Why microservices here
Most apps do not need microservices. This one does, because the workload is genuinely lopsided: the API is cheap and constant, the conversion is expensive and bursty. Splitting them means you can run one API instance and ten converters during a spike, then scale back down. The queue is what makes that possible — it decouples 'work requested' from 'work performed'.
The service map
Four services: an API gateway that is the only public entry point, an auth service that owns identity and issues tokens, a converter service that consumes queue messages and runs the transcoding, and a notification service that tells the user when their file is ready. None of them call each other synchronously for the heavy path.
Storing large files
Video files exceed the practical row size of a normal database. MongoDB GridFS chunks large binaries across documents, so uploads and results are stored and streamed back without holding an entire file in memory. Relational data — users, jobs, statuses — stays in PostgreSQL where it belongs.
One repo, many services
It is a monorepo: shared types, shared lint config, one command to spin the whole system up with Docker Compose, and end-to-end tests that exercise a real upload through a real queue rather than mocking it.
Section 04
The build process
Find the bottleneck
I profiled a naive single-service version first. The conversion step dominated everything, which justified the split — I did not start with microservices as an assumption.
Define message contracts
Before splitting, I wrote the message payloads. Contracts on the wire are the real API in a queue-based system, so they got designed deliberately.
Extract services one at a time
Auth first, then conversion, then notifications. Each extraction kept the system running so I never had a big-bang rewrite to debug.
Containerise and orchestrate
Every service ships as a Docker image with health checks, then Kubernetes manifests define replicas, resources, and restart behaviour.
Prove it end to end
E2E tests upload a real file and assert the MP3 comes back, exercising the queue and both databases rather than stubs.
Section 05
How the pieces fit together
Each layer has one job. Read it top to bottom — that's roughly the path a request takes through the system.
Next.js frontend
Upload UI with job status polling and download of finished audio.
API Gateway (NestJS)
Single public entry point. Validates requests, checks auth, publishes jobs, never blocks on conversion.
Auth service
Owns users and tokens so no other service handles credentials.
RabbitMQ
Durable queues decouple request from processing and provide retries and backpressure for free.
Converter + Notification services
Independently scalable consumers. Conversion runs off queue depth, notifications fire on completion events.
PostgreSQL + MongoDB GridFS
Relational data for jobs and users, chunked binary storage for the media itself.
Section 06
What it does
Non-blocking uploads
The request returns as soon as the file is safely stored and the job is queued.
Independent scaling
Add converter replicas without touching the API tier.
Automatic retries
A worker crash re-queues the message instead of dropping the job.
Completion notifications
A dedicated service watches for finished jobs and notifies the user.
Section 07
The hard parts
Message idempotency
Queues guarantee at-least-once delivery, so a job can be delivered twice. Consumers check job state before working to make reprocessing harmless.
Local developer experience
Six moving parts is hostile to onboarding. Docker Compose brings the entire stack up with one command and seeded data.
Observability across services
A failure in a distributed system is hard to locate. Job IDs propagate through every message so a single upload can be traced across all services.
Section 08
What I took away
- Queues are worth it when workloads have different scaling shapes, and overkill when they don't.
- At-least-once delivery makes idempotency a requirement, not an optimisation.
- E2E tests against real infrastructure catch what unit tests structurally cannot.
Architecture
Microservices
Messaging
RabbitMQ
Storage
MongoDB GridFS
Orchestration
Kubernetes
Let's build something worth shipping
Available for freelance projects, full-time roles, and technical consulting. Tell me what you're building and I'll tell you honestly whether I'm the right person for it.
Based in Rio de Janeiro, Brazil · Working with teams in any timezone · Fluent in English and Portuguese