TernFS – An exabyte scale, multi-region distributed filesystem
8 comments
·September 18, 2025mrbluecoat
Cool project and kudos for open sourcing it. Noteworthy limitation:
> TernFS should not be used for tiny files — our median file size is 2MB.
heipei
Yeah, that was the first thing I checked as well. Being suited for small / tiny files is a great property of the SeaweedFS system.
pandemic_region
What happens if you put a tiny file on it then? Bad perf, possible file corruption, ... ?
jleahy
It's just not optimised for tiny files. It absolutely would work with no problems at all, and you could definitely use it to store 100 billion 1kB files with zero problems (and that is 100 terabytes of data, probably on flash, so no joke). However you can't use it to store 1 exabyte of 1 kilobyte files (at least not yet).
redundantly
Probably wasting space and lower performance.
ttfvjktesd
How does TernFS compare to CephFS and why not CephFS, since it is also tested for the multiple Petabyte range?
rostayob
(Disclaimer: I'm one of the authors of TernFS and while we evaluated Ceph I am not intimately familiar with it)
Main factors:
* Ceph stores both metadata and file contents using the same object store (RADOS). TernFS uses a specialized database for metadata which takes advantage of various properties of our datasets (immutable files, few moves between directories, etc.).
* While Ceph is capable of storing PBs, we currently store ~600PBs on a single TernFS deployment. Last time we checked this would be an order of magnitude more than even very large Ceph deployments.
* More generally, we wanted a system that we knew we could easily adapt to our needs and more importantly quickly fix when something went wrong, and we estimated that building out something new rather than adapting Ceph (or some other open source solution) would be less costly overall.
Wow, great project.