How Accurate is Flow? Follow
The short version
Flow is an air quality monitor for the rest of us. It was designed to take pollution measurement out of the lab and into the public’s hands. Given the technical constraints this mission adds to the design of such a device, Flow is best-in-class for accuracy and reliability over time.
In scientific terms, Flow is at 90 to 95% correlation with static reference monitors in benchmark tests for the core pollutants we measure.
It was developed by a team of air quality and electronics experts in collaboration with renowned scientific and academic partners around the world and is designed to be the air quality device best suited to personal health applications.
The long version
There are a few ways in which the question needs to be framed: Like being asked “how fast is a train?” it’s tough to answer without a few follow-up questions.
Accurate compared to what?
A good first step would be to break the air quality (AQ) monitoring equipment space down into simple categories, which might look something like this:
Monitoring stations: Used by local or national authorities (there are 12 stations inside the walls of Paris, France, for example) to quantitatively inform policy decisions and prove compliance with international regulations. They’re as precise as AQ sensors get and can cost anything between $30k-$150k to install, and require maintenance and a dedicated team. Needless to say, you’d have a hard time setting one up in your backyard...
Lab monitors: Mostly used in… labs! They’ll vary somewhere between the size of a desk and the size of your two stretched hands, and will tend to have at least three zeros on the price tag. They’re very precise, but bulky and designed to work in controlled environments (where, as the experimenter, you define and control what’s in the air). Our own R&D engineers use these to test and calibrate Flow.
Consumer-grade monitors: They’re designed to be used by everyone and anyone. They sit in a living room or in the palm of your hand. Their goal is to measure air quality in your everyday life (which is definitely not a controlled environment). In essence, consumer sensors and especially mobile ones are an attempt at miniaturising monitoring station technology into a handheld format for down to one-thousandth of the cost. As you can imagine, this means some compromises will have to be made… we’ll explore some of them below!
What accuracy should I expect from a consumer-grade monitor?
At a hundredth or a thousandth of the manufacturing cost of a government-funded monitoring station, expecting perfectly matching readings from a station and a consumer-grade monitor in a side-by-side comparison wouldn’t make a whole lot of sense.
Consumer-grade monitors are designed with personal health in mind, so this means they should be really good at:
- Helping users understand the pollution levels they are exposed to in terms of thresholds corresponding to different health risks.
- Providing context and insights to help users understand their exposure in relation to their environment’s average as well as the rest of the population’s average.
- Accurately detecting variations and peaks - from a personal health perspective, consistent, dependable and real-time detection of air quality suddenly changing thresholds is on the very top of the priority list.
What does a focus on the detection of variations and peaks mean?
It is the result of the tradeoffs between device size and ability to reach lab-grade precision that physics imposes on AQ monitors.
Measuring particulate matter (PM) concentrations for example is done using a laser - simply put, the more powerful the laser, the more precise it will be. However the more powerful the laser, the bigger it is and the more energy it consumes. Very quickly along that scale, making a handheld device that is as accurate as a government monitoring station no longer becomes possible, especially if its battery is to last more than a few minutes.
From a daily use perspective, the way the tradeoffs between accuracy and size translate is comparable to the way we use consumer-grade thermometers: A deviation of plus or minus 0.1 or 0.2°C is acceptable because what really matters is precision with the order of magnitude above that: I’m not going to worry if my temperature goes from 37.0 to 37.2°C but I am going to worry if it goes from 37 to 39°C.
With air quality, what we work with is detection ranges between 0 and 2000 parts per billion or 0 and 200 micrograms. The complexity involved is several orders of magnitude above that of thermometers. However we know users expect no less :)
So, the team behind a good consumer-grade air quality monitor will first and foremost have invested its energy into extremely reliable detection of the movement of ambient air quality between health-impact-related thresholds.
Why is Flow a best-in-class consumer-grade air quality monitor?
In essence, Flow is the combination of two things:
- Electronic components (sensors, fans, chips, resistances, all soldered onto a circuit board) that turn a constantly changing amount of particles and molecules in the air into a variation of electrical current
- Firmware, or the “brain” of the device, that translates the variations of electrical current into a digital signal. This signal is then interpreted into pollutant concentrations that are in turn converted into an AQI and beamed to the Flow app over Bluetooth
Combined, Flow’s electronics and firmware position the device to best work around the accuracy constraints inherited from its portable size.
For more on this and how Flow works in general, you can check out this blog post
Flow, the device you are holding in your hand (or thinking of holding in your hand), is the result of four years of R&D by a team of atmospheric sciences and electrical engineering PhDs, data science and machine learning experts, industrial design and manufacturing veterans.
This time and expertise brought together has enabled us to create a device that:
- Measures the single widest array of pollutants that matter most to our health both indoors and outdoors: PM2.5, PM10, VOCs and NO2, providing 24H of continuous readings on a single charge (see more on Flow’s battery life here)
- Self-calibrates with cutting-edge machine-learning algorithms contained in Flow’s proprietary firmware (see next section)
- Contributes to a worldwide, crowd-sourced and real-time mapping of air quality (see this blog post for more information)
Each new firmware update brings improved accuracy and calibration capabilities. It is systematically and thoroughly tested in collaboration with our scientific and academic partners, a few of which are the LISA lab at the CNRS in France, King’s College and Imperial College in London and Sheffield University. Because a major part of the team behind Flow comes from a scientific and/or academic background, it is essential to us that everything we do reaches the highest standards that can be set for a consumer-grade monitor.
For a detailed, statistical view of where Flow stands, you can find a copy of our white paper from October 25th, 2019 on Flow’s accuracy by following this link. Its main finding is that Flow is at 90 to 95% correlation with static reference monitors in benchmark tests for the core pollutants we measure.
How does Flow’s accuracy evolve as the device ages?
In the world of electronics, there’s a concept known as drift that applies unforgivingly to any and every electronic device.
Let’s take the example of one of the most common electronic components you can find on a circuit board, a resistance, of which Flow houses 73. Its job is to diminish the electrical current it receives by the value indicated by its markings. However, every resistance has a tolerance, i.e. an accepted slight deviation against its “advertised” resistance: In reality, no two resistances are ever exactly the same, with deviations that can be from 0.01% all the way up to 5%... So when you have 73 resistances on your circuit board, the deviations can start adding up.
But it doesn’t stop there. Over time, electronic components age and their chemical and mechanical properties will be slightly altered. Meaning a resistance, for example, will deviate further from its nominal value as time goes by. When you apply this to all of the components of a sensor whose job is to measure and translate variations in electrical current, you begin to understand where the problem can lie. This is known as drift.
The solution is calibration – the act of adapting the algorithm you apply to interpret variations in electrical current to account for component drift.
Many consumer monitors will ask you to perform manual tasks for calibration to happen. Flow, on the contrary, is self-calibrating thanks to its firmware: Calibration happens automatically in the background every single time your device and app synchronise over Bluetooth.
What this means is that without the user having to perform any device maintenance whatsoever, Flow automatically guarantees the stability of its own measurement capabilities over time.
Will several Flows side-by-side give me the exact same measurements?
If you’ve made it this far (thank you and high five, by the way!), you’ll likely have an intuition of what the answer could be…
If you sit two or three or ten Flows side by side and compare the concentrations and AQI they’ve measured, will you see deviations? Yes, that’s likely - because of everything we’ve already explained, the contrary (absolutely 0 AQI deviations on 4 pollutants between 10 devices) might even be slightly suspicious.
What you should expect from several Flows in the same room is that they consistently position themselves within the same pollution threshold.
With a nuance: Say for example that the “real” AQI in the room is 19 on the Plume Labs scale - because 20 PAQI is the limit between Fresh Air and Moderate Pollution on the Plume scale, within acceptable deviation, it’s quite likely some Flows will show “Fresh Air” and others “Moderate Pollution”. What should not happen, however, is that any of them display “High” or “Very High” pollution. If ever that happens, please contact support)
What can I do to maximise my Flow’s accuracy?
Quite simply: Use it as much as possible.
Two main reasons behind this:
- Dynamic calibration: if long intervals of time are left between device-to-app synchronisations, Flow’s firmware is going to have a much harder time keeping measurements accurate. We currently recommend at least one synchronisation per day.*
- The more active the Flow community is, the more feedback and data we can work with to improve Flow’s accuracy and calibration even further!
In conclusion
- Flow was built to be road-worthy, for the streets but not for the labs. As such, the advances in electronics and self-calibration Plume Labs was able to achieve make it a truly best-in-class wearable device in many respects, accuracy included.
- What matters most for Flow to accomplish its mission is that it excel in faithfully measuring and displaying variations in air quality that affect our health. Flow version 1 is already showing extremely strong correlation when benchmarked against lab-grade equipment, and will only keep getting better.
- Flow’s accuracy over time is linked to its usage because of on-board, automatic calibration. The more you use Flow, the more accurate it remains.
*For some of you, that may seem like a lot, which is why we will be delivering dynamic calibration on-device very soon (no app synchronisation required for calibration)!
Comments
1 comment
Thank you for writing such an extensive explanation!!
Please sign in to leave a comment.