@DrSimonClark some cyclers record time as an integer number of milliseconds internally, Newares also do this. My opinion is that seconds feel more natural, and I don’t think the precision loss storing as a float matters.
It would be nice if there was a way to separate label and pint-compatible unit in the column names. e.g. if units were put in square brackets or separated with a different character like double underscore, then tools that read bdf can handle any units with pint. I don’t know if that makes the ontology part impossible.
Some cyclers store time as a millisecond integer to avoid accumulation of errors due to floating-point precision issues. I think this is more important during the measurement itself, when errors can accumulate over a long time. But for serializing to a global interoperable format, it’s best to stick to community-standard units and the precision of converting integer to decimals should be ok.
Handling of units is also an important consideration. That’s why we’ve built in lots of support to parse and convert units. These will be included in a python package that we’ve been working on. Check out the units notebook here. As you mention, we build on top of pint to easily handle unit conversions in python. On the simplest level, you can just strip the unit from the preferred label or the machine name and use that in pint directly. We also have units encoded in the application ontology and the table schema for semantic operations. So hopefully this will provided robust redundancy to handle units. What do you think?
Hi Simon, a bdf python package would be great, the validator would be particularly useful for other devs (me) trying to ingrate bdf in our pipelines.
I wouldn’t immediately know how to strip units, as there is no reserved character for separating labels and units in names, e.g. charging_capacity_milliampere_hour - where does the label end and unit start? But if a bdf package can do all this for me I am happy.
pint-pandas may be of interest for you if you haven’t seen it yet.
One schema decision worth making early is how to represent time and test events. A timestamp should say whether it is absolute or relative to the test, and step transitions should be explicit rather than inferred from current, voltage, or a vendor’s step index. That would make mixed-rate data and comparisons across cyclers much easier to interpret, even when implementations store the records differently.
A useful baseline would be to define units and sign conventions explicitly, rather than relying on cycler defaults. For each measurement, the schema could distinguish the instrument’s recorded value from any normalized or derived value, and preserve both the timestamp and elapsed time. Those details matter when comparing datasets from different vendors or reconstructing a test sequence.
The schema should distinguish raw observations from values calculated later, with provenance for each derived field. For every measurement, define the quantity, unit, timestamp basis, and whether the value is instantaneous or integrated; otherwise two files can look compatible while reporting different meanings for “capacity” or “time.” A small set of validation examples would make those rules testable across implementations.
A minimum record should define not just the measured value, but its unit, timestamp basis, and measurement channel. In particular, elapsed time versus wall-clock time and whether timestamps mark sample acquisition or step transitions can change downstream analysis. Making those semantics explicit would help preserve compatibility without prescribing a file format or cycler implementation.