We’ve been running into a practical challenge while preparing some larger datasets for release, and I’d love to hear how others in the community are handling this.
When a dataset grows beyond a certain number of files, platforms like Zenodo automatically zip the upload. This is understandable, but it breaks the folder-based structure that BDF encourages and makes it harder for people to browse, inspect, or load subsets of the data without downloading everything.
It’s a small technical issue in the grand scheme, but it has real implications for usability, especially once groups start publishing full test campaigns, long time-series, or high-resolution metadata.
Questions for the community:
Where are you hosting your large datasets today?
Have you found platforms that preserve folder structure without zipping everything?
Is anyone using Git-LFS, S3, HuggingFace Datasets, Dataverse, Dryad, or institutional repositories?
Would a “BDF-friendly” hosted storage solution be valuable?
Are there best practices we should document for dataset authors so the ecosystem converges on a few stable options?
We’d really appreciate hearing your experiences and recommendations. If there’s interest, we can compile everything into a community-driven guide for BDF dataset hosting.
Most datasets today seem to be hosted on platforms like Zenodo, Mendeley Data, or institutional-specific repositories. There are a few benefits to those, e.g. (i) they are free, (ii) they mint DOIs for you, and (iii) they handle the bibliographic metadata management.
But, they can also be difficult because they have limits on the number of files per record (e.g. 50 GB and 100 files on Zenodo) and there is no uniformity in how records are structured.
Git-LFS might be suitable for moderately sized records (e.g. < 1 GB) and it offers an archiving service with DOIs through Zenodo. But I’m not sure how well it scales to 100 GB - 1 TB+ (both from a cost and performance perspective).
I’m exploring options like S3 and R2 for hosting the raw data files with an optional git layer for version control.
For very large campaigns, an object store with public HTTPS access and range requests could keep individual files browsable and let users fetch only what they need. A DOI repository could still hold the citation metadata and a small manifest pointing to the data, though the long-term availability of those external URLs needs to be addressed. A BDF hosting guide should specify a machine-readable file listing, checksums, versioning, and a clear policy for moving or replacing files.
A useful compromise is to separate discovery from storage: publish a DOI-bearing record with the BDF directory tree and a manifest, while keeping the large files in object storage. The manifest can include checksums, sizes, and stable object keys, so users can verify files and fetch only the subset they need. The record should document access limits and retention terms, since an S3-compatible endpoint alone doesn’t guarantee long-term availability. Git can version the manifest and metadata without putting the raw dataset in Git-LFS.
Coming from video hosting rather than battery testing, so take this with a grain of salt, but the storage side is very similar.
One thing worth checking before settling on S3: egress cost. Once a popular dataset gets pulled a few hundred times, the download bill can easily dwarf the storage bill. R2 or Backblaze B2 behind a CDN avoid most of that, which matters for a community project without a fixed budget.
I’d also second the manifest idea, and suggest keeping files immutable: a new version gets new object keys and a new manifest, and the old files are never overwritten. That way a DOI always points at exactly the bytes people cited, and caches never serve a stale file under the same name.
A useful split is to keep the DOI-bearing record as the citation and metadata layer, with the large files in object storage. Publish a manifest containing relative paths, sizes, and checksums alongside the BDF directory tree; that makes partial downloads and integrity checks possible even if a repository packages the files into an archive. For long-term access, the guide should also say who pays for storage and how a stable copy or migration is handled, since S3-compatible storage alone doesn’t guarantee preservation.
An object store can keep files individually browsable, with a small landing page that carries the DOI and citation metadata. Put a machine-readable manifest beside the data with relative paths, sizes, checksums, and schema/version info; that gives users a way to validate downloads and select subsets without pulling the whole campaign. Keep published prefixes immutable, and publish a new version rather than silently replacing files.