MpegFlow with MinIO: self-hosted S3-compatible video storage

MinIO is the open-source, S3-compatible object store you run on your own hardware. For sovereign-cloud, regulated, or scale-economic video workloads where AWS S3 isn't an option, MinIO is the production-grade replacement. MpegFlow's integration is identical to S3 — same presigned-URL pattern, same strict-broker security, same SDK.

How the integration works

MinIO implements the S3 API. MpegFlow's coordinator points at MinIO endpoints (e.g., https://minio.example.com:9000), generates presigned URLs for jobs, workers download / upload via standard S3 SDKs. The pattern is the same as cloud-resident S3; only the endpoint changes. Distributed MinIO clusters (4+ nodes with erasure coding) handle production-scale video storage with no AWS dependency.

Common patterns

• Air-gapped video pipelines

  For workloads requiring network isolation (defense, regulated healthcare, certain studio pre-release content), MinIO + MpegFlow self-hosted runs entirely in your perimeter. No outbound calls to cloud services. The license-JWT validation works offline; the encoder pool runs against MinIO; the audit log writes to local Postgres.

• Erasure coding for archive

  MinIO's erasure-coding (default EC:4 across 4-8 disks) provides storage durability comparable to S3 at on-prem hardware costs. For petabyte-scale archives, this beats both AWS Glacier (egress costs) and dedicated archive vendors (operational cost) once you're committed to running storage hardware.

• Edge ingest with central encode

  For broadcast workflows with on-site contribution (live remote events, location production), MinIO clusters at the edge collect ingest, replicate to a central MinIO cluster, and the encoder pool runs against the central. The pattern reduces contribution-protocol fragility because contribution is to local storage, not cross-internet to cloud.

• Hybrid cloud + MinIO

  Many production deployments mix MinIO (on-prem mezzanine + master archive) with cloud storage (delivery + transient encodes). MpegFlow's coordinator handles arbitrary endpoint configurations — different jobs route to different storage backends based on customer or workload class.

Pitfalls

1. MinIO scales linearly with hardware — capacity planning is on you, not auto-scaling. Plan disks and bandwidth carefully for projected growth.
2. MinIO's S3 compatibility is high but not absolute — certain newer S3 features lag (Object Lock variants, some IAM scenarios). Test specific workflows.
3. Operating MinIO at production scale requires real SRE capacity: monitoring, backups, version upgrades, hardware lifecycle. AWS S3 abstracts all of this.
4. Network throughput between MinIO and encoder pool dominates performance — plan 10GbE+ between them, especially for 4K/HDR workflows.
5. MinIO licensing: AGPL v3 for the open-source version. Commercial license available from MinIO Inc for closed-source deployments. Talk to legal.

At production scale

MinIO at production video scale typically lands at 5-20 PB clusters spanning 50-200 nodes. Hardware cost (drives + servers + networking) is the dominant capital expense; operational cost (electricity, cooling, SRE time) is the dominant operating expense. For workloads where AWS S3's per-GB-month cost is unaffordable (multi-petabyte archives accessed frequently) or sovereign-cloud is a hard requirement, the math works. Below ~1 PB, AWS S3 is operationally simpler and economically comparable.

• minio
• Self-hosted
• storage
• integration
• sovereign-cloud