Open standard for cycler output data

[smedegaard]

Great to hear @gabe

I have had some time to think about ways to proceed on the data structure.
My general thinking is to use a data format that supports nested data types to hold the meta data of a given battery test. That is the json part of VDF.
Depending on the implementation, the test data (the csv part) would be inside a nested structure as well (depending on the db/file format, often called a struct)

A good first reference implementation could be using the Delta Lake format.

Inviting the folks from Voltaiq to this thread is a good first step.

[valentin]

What are people’s thoughts on opinionated vs flexible column names in the file format?
Opinionated: you have to call the voltage “Voltage [V]” and its units have to be volts
Flexible: projects have a config file that specifies these things: {VOLTAGE_NAME: “Voltage”, VOLTAGE_UNITS": V}

My thoughts are that it depends if we care more about the standard or the data management software, but if we care about the standard, we should be opinionated. If we care about the data management software, we should be more flexible. Being flexible would allow us to read in data from a variety of other public sources without needing to convert and store it in our n+1th standard. But it might makes things unnecessarily complicated in practice

[smedegaard]

Good question @valentin,

As you say, it depends on what we are solving for.

In my mind it’s about managing and working with large amounts of test data created by cyclers and their software.
I don’t see how we can accomplish that without defining some structure for the data. It can have some flexibility in terms of evolving the schema, but my worry is that manually creating configuration file may corrupt reading the data.

But I don’t have lab experience so I might be missing some context from how the equipment and software is used in the wild.

[smedegaard]

Hi all.

I’ve given it a little thought and I think a usable solution could be a nested structure in any data format/database that supports it.

The big question for me is if it’s too unfamiliar for people to work with.

I’ve made a notebook that only defines a data schema for now. I will expand with some dummy data, and see how well it performs when reading and writing large amounts of data.

[gabe]

The data schema you defined, does your example leverage the VDF? And the battinfo battery ontology?

[smedegaard]

Naming of the fields is inspired by vdf. There is no ontology integration yet.

[chintanp]

Adding to the discussion, BattGenie has already open-sourced data ingestion python package - BattETL - that works with Maccor and Arbin data, and a database to store data as well as metadata - BattDB.

BattETL: GitHub - BattGenie/battetl: A module for extracting, transforming, and loading battery cycler data to a database.
BattDB: GitHub - BattGenie/battdb

We will soon be releasing, a FOSS version of both above under the banner of BDA, which will also include a visualization layer for the ingested data.

Let me know if there any questions about the above.

[gabe]

Here is the video from our meeting discussing how to leverage Battery Ontology with BattETL & BattDB.

Thanks @DrSimonClark for the demo!

[gabe]

Continuing this thread, there’s since been quite a bit of progress including work on the Battery Data Format.

Thanks @DrSimonClark for publishing the first cell in the BDF to Battery Knowledge Base Discharging Time Series of a CR2032 Battery at 11 mA

This really shows the power of the Battery Ontology and aligning with best practices and shows the possibilities of what these tools can do. Looking forward to see what this will enable.

We recently had a great conversation with folks from Faraday Institute who are working on a similar problem and their solution is largely aligned. After some discussion, the next BDF proposal would be to align to 4 required fields (if we adopt cycle_dimensionless → step_count)

• test_time_millisecond
• step_count
• current_ampere
• voltage_volt

With the following open questions remaining:

1. When to increment step? is it aligned to the program step and can repeat or is it increasing by 1 and never decreasing? or is it aligned to software protocol?
2. Time measured in seconds vs milliseconds
3. Naming convention, how to include units in the name. The choice to use _unit was to ease machine readability and parsing for unit conversions

Interested to know what folks think of the above.

[DrSimonClark]

1. When to increment step? In my opinion, it should be monotonically increasing anytime there is a change in the control mode.

2. Time measured in seconds vs milliseconds? Second is more intuitive and user-friendly than millisecond. I suspect millisecond was chosen because it is more likely to be an integer. But I don’t have a strong opinion about it.

3. Naming convention for units? The requirements that I see are that they should be (i) easily parse-able, (ii) explicitly clear, (iii) align to an existing standard. Plain symbols for units open up case-sensitivity issues (e.g. s and S) - so the actual name is more explicit/robust (e.g. second and siemens). Pint offers a good package for that, doing unit conversions, and mapping to unit ontology terms. That’s why this style was recommended.

[smedegaard]

millisecond vs. second, etc.
In general I would just advice to not truncate the values early on. Numbers can always be formatted to be human readable. But decimal point values may carry information for machine learning and analysis

about units: a more robust way than putting the unit in the field name would be adding a data structure just for units like in the notebook I shared earlier.
In python that could look something like

@dataclass
class SIUnitDataMixin:
    sign: str
    unit: int
    dimension: str

class SIUnit(SIUnitDataMixin, Enum):
    AMPERE = ("A", "ampere", "electric current")
    AMPERE_HOUR = ("Ah", "ampere-hour", "electric capacity")
    CELCIUS = ("c", "celcius", "temperature")
    VOLT = ("V", "volt", "electric potential")
    MILLISECOND = ("ms", "millisecond", "time")

    def to_dict(self):
        return {
            "sign": self.sign,
            "name": self.unit,
            "dimension": self.dimension
        }

Example

SIUnit.AMPERE_HOUR.to_dict()
{'sign': 'Ah', 'name': 'ampere-hour', 'dimension': 'electric capacity'}

Depending on the database or type of files one uses for storing the data a measurement could either

point to a table of SI units in a relational database:

or have the unit embedded a la

{
...,
"measurements" : [
  {
    "value": 42.42,
    "unit": {sign: "A", "unit": "ampere", "dimension": "electric current"}
  },
  {
    "value": 3.14,
    "unit": {sign: "A", "unit": "ampere", "dimension": "electric current"}
  },
].
...
}

[DrSimonClark]

wrt units: putting it in the field name is mostly just a helpful flag for the humans. The real work is handled in the metadata.

What you describe @smedegaard is the functionality that we want, but we’d rather build on existing infrastructure than create something new. Unit definitions, conversions, and quantity validation have been handled extensively in resources like pint, qudt, and emmo.

The metadata for the BDF table schema handles the declaration of units for the machines. But if a human wants to parse the key to work with the unit in pint, they can do that.

[TomHolland]

Thank you for the invitation to contribute to this discussion @gabe.

On the format of units:

I think my perspective (from developing the user-interface layer to a battery data format with PyProBE) is aligned with a point I see Valentin made in this thread before. I appreciate the motivation for avoiding ambiguity that a verbose spelling of the unit achieves, but I see a couple of drawbacks:

1. Ease of use and user acceptance- I would suggest it is ok for a standard to be opinionated, and the SI symbol standard is well established so distinguishing between upper and lower case S for seconds and sieverts seems like a relatively minor problem. In contrast, spelling out the unit can get convoluted especially with combination units. An extreme example of this would be heat flux (watts-per-metre-squared?). While compound units like these are relatively rare in time-series battery data, they are extremely common in the parameters of battery models which a researcher may well be computing from the data stored in the bdf format. It would be nice I think for the human-readable string form of these quantities to remain consistent when passed from raw data, to processing tools and finally to models.
2. A more concise unit format can be also plotted directly from the data source, for example with pandas df.plot(“Time [s]”, “Voltage [V]”) while a more verbose form would require manual override of the axis labels. While this would be very simple, it may be off-putting to a user. It also would mean that the standard would be obscured when the data is presented in published work.
3. Finally, the unit names are not without ambiguity- spelling varies in different languages, and even in British and American English: metre vs meter!

I 100% agree with using existing tools like pint for user interaction with units (my current implementation of units in PyProBE works for common battery units, but is very basic and not scalable so I have it on my to-do list to integrate pint), but as the column names are user-facing, conciseness and simplicity is important.

I can see that there are a few different ways that units are described in the table schema on top of the column name, I assume these are each serving separate purposes like linking to other standard (emmo for instance)? Are there parallels with this metadata field and something like “known aliases”, where the different column names and unit styles of battery cyclers could be recorded e.g. “biologic:Ecell/V”? Sorry if this is the wrong end of the stick; this may be better suited to a database that links the standard to the multitude of other formats that exist. My implementation is a set of classes that can convert column names and compute derived columns (capacity from charge and discharge capacity for example), but I see maintaining the various cycler format parsers as a significant ongoing maintenance challenge so was curious to hear your thoughts.

On the definition of step:

I think it is most useful if it is aligned with program step for identifying sections of an experiment. It then naturally also includes cycle information, which can be identified from when the step number decreases I.e. Step = [1,2,3,1,2,3], Cycle=[1,1,1,2,2,2]. It isn’t clear to me why you would ever remove this information, if your cycler has recorded it, in order to make this column monotonically increasing. I don’t think anything is lost by allowing step numbers to repeat- if the data source provides repeating step numbers then individual program instructions can be separated, and if not the behaviour would fall back to a monotonically increasing count.

[DrSimonClark]

wrt units:

The column name is for human use and readability, and so ease of use and user acceptance should be prioritized. That comes down to a stylistic choice.

Using a formatted label, like “Voltage / V” is acceptable, as long as the symbol for the unit can be parsed by existing tooling like pint to provide an extra layer of robustness.

• This has the pro that it is nicely formatted for human-readability.
• It has the cons that it includes capitalization, spaces, and special characters - which all present opportunities for mistakes.
• It could also have formatting problems for units with superscripts like W/m² .
• In my opinion, “voltage_volt” is more human-friendly because it has fewer opportunities for mistakes, even though it is more verbose.

If we do go with formatted labels, then we should note that the SI and IUPAC best practice for quantity notation is to use a slash rather than parenthesis or brackets (e.g. “Voltage / V”) - because quantities follow the rules of algebra. (covered by section 5.4.1 of the 2019 SI Guide or section 1.1 of the IUPAC Green Book).

The main thing is that the units are robustly handled in the metadata, which is covered.

Remember that we can define all kinds of labels in the metadata using rdfs:label or skos:altLabel or skos:hiddenLabel. Known aliases are compiled in the parent class terms from the electrochemistry domain ontology (e.g. current), which can be queried in the event that a table schema is not available.

[DrSimonClark]

I’ve been asked to record a lecture series on 5-Star Battery Data as a resource for EU projects. It’s a good opportunity to include BDF as part of it, if we can settle on what labels we’d like to prioritize.

We can make it robust in the metadata to cover all known aliases for header matching with csvw:title , but we should still pick one to promote as the “canonical label”.

"csvw:name": "voltage_volt",  
"csvw:title": ["Voltage / V", "Voltage [V]", "Voltage (V)", "voltage_volt", ... ],  
"csvw:propertyUrl": "bdf:voltage_volt"

I’m coming around to the idea that the user-facing label should be in the SI / IUPAC style convention for humans Voltage / V. I know that square brackets Voltage [V] are popular and we can include them as a hidden alias, but I’d rather not make them the canonical label since they contradict SI and break the logic of quantities.

For complex units, the heavy lifting is done in the metadata mapping to the term in emmo and qudt - but this can be expressed in unicode such that the label is still SI compliant and still processable in pint as a fail-safe, Specific Capacity / mA·h·g⁻¹ and the BDF IRI would be bdf:specific_capacity_milli_ampere_hour_per_gram.

@gabe, what are your thoughts?

[TomHolland]

Thanks all for the enjoyable discussion yesterday.

On units
What you say above @DrSimonClark makes sense to me. Presumably avoiding special unicode characters would also be valid if written in a way processable by pint i.e. Specific Capacity / mA*h*g^-1.

On definition of step
I think the current definition of StepIndex is actually correct for what I was trying to explain in the call:

StepIndex: the currently running step number in the procedure

This should correspond directly to the program written by the cycler:

This is an example from Neware, the resulting StepIndex column would only contain numbers from 1-13 and would repeat numbers from 4-7 and 9-12 because of the cycle instructions.

In PyProBE, cycle number is then inferred when the StepIndex decreases i.e. when it decreases from 7 to 4 in the first cycle. Alternatively you can explicitly break up the Procedure into named sections and specify cycles (allowing for nested loops) in the .yaml file I mentioned:

Initial Charge: # Experiment title
  Steps: # Steps list, descriptions are PyBaMM experiment strings
    1: Rest for 4 hours # Start with index 1
    # CC-CV charge provided as two seperate PyBaMM experiment strings
    2: Charge at 4mA until 4.2 V, Hold at 4.2 V until 0.04 A 
    3: Rest for 2 hours
Break-in Cycles:
  Steps: 
    4: Discharge at 4 mA until 3 V
    5: Rest for 2 hours
    6: Charge at 4 mA until 4.2 V, Hold at 4.2 V until 0.04 A
    7: Rest for 2 hours
  Cycle: # Cycle instruction, contains start, end and count
    Start: 4 # loop starts with step 4 (inclusive)
    End: 7 # loop ends with step 7 (inclusive)
    Count: 5

This then leaves defining the continuously increasing step count:

StepCount: a count of the number of steps since the beginning of the procedure

I’ve come around to this continuously increasing value being the column specified as the “bare minimum”, as long as StepCount and StepIndex are separately defined.

Further to this, when defining a standard it may be important to avoid the misconception that the four required columns are an acceptable reduction from detailed cycler data. There are plenty of columns that cyclers write out that are redundant e.g. all the different variations of Capacity (“Charge”, “Discharge”, “Step” etc.) which can be derived very simply from a single Capacity column which could be defined as:
NetCapacity: the net charge passed into a cell since the beginning of a procedure

However, there are columns that, if present in data retrieved from a cycler, shouldn’t be removed:

• StepIndex (as defined above) - as it is more directly useful than StepCount as I have described before
• NetCapacity - as cyclers coloumb-count internally faster than they log data so this will be more accurate than integrating current with time

Perhaps these could be added to the recommended columns to store alongside DateTime?

[gabe]

@DrSimonClark On labels: I agree that having an SI-style label as the preferred or default form makes sense in terms of aligning to international standards, but we should remain permissive to other conventions like [] or () that are common in different communities. Supporting them as aliases keeps things interoperable.

@TomHolland On step metadata: Glad we’ve landed on a definition that works. It makes sense to separate StepCount and StepIndex . The BDF is meant to be a format of last resort — permissive and able to support richer metadata when available, but only requiring the core set. We don’t recommend or require users to remove columns in order to be BDF compliant. As we expand supported analyses with BDF, we can introduce linting or validation to detect whether the necessary columns are present to enable a given analysis.

[DrSimonClark]

@TomHolland , would you clarify the definition of NetCapacity? The word Net implies that the value is adjusted to compensate for some other contribution. Is it the difference between the ChargingCapacity and the DischargingCapacity (i.e. cycling up and down from 0) or only the charge passed into the cell (i.e. TotalChargingCapacity)?