Valencell evaluates a lot of wearables in our Biometrics Lab. Last year we completed over 36,000 device tests measuring over 2,000 hours of data collection and validation on biometric wearables. Almost all of this testing uses an electrode-based (ECG) chest strap to validate the performance of the device or devices we’re testing. Most people consider ECG chest straps to be the gold standard for testing biometric wearable devices and for good reason. They are very accurate for measuring heart rate and R-R interval. However, there’s a misconception in the market that needs to be corrected:
The misconception is that ECG chest straps work all of the time. The reality is they are not 100% accurate. Failures of tracking metrics occur just like any other biometric wearable device.
With over a decade of evaluating all kinds of evaluating these devices, we have learned that the amount of mistracking in ECG chest straps is not an insignificant number. To provide a baseline for these devices we set out to quantify their accuracy by measuring two chest straps, the Bluetooth low energy Polar H7 (BLECS) and Polar CX800 watch and chest strap (CSHRM), on the same subjects determine the frequency of data points within +/-5% of each other (data measured once per second).
Figure 1
Each participant completed the standard “Valencell Test”, a single, dynamic, 8-minute treadmill session consisting of standing, walking and running in order to elicit a variety of exercise intensities. Here are the details of the protocol:
Results
It turns out that the two chest straps agree with each other within +/-5% across 92% of the data points. Here are the detailed results:
So even the gold standard has a bad test every now and then, just like every other device. The internal processes of the human body are difficult to measure, particularly during movement and exercise.
If you’re wondering about data you may have seen that shows chest straps are 99-100% accurate compared to an ECG, those data points are likely NOT including data and/or tests where there are obvious errors in the data. In data sets where the chest strap is obviously not tracking to the trained eye one can clearly see when a chest strap is providing inaccurate data. This typically comes in the form of highly erratic data, which are physiologically unlikely in a healthy individual during activity, including highly variable heart rate “jitter” or reported heart rate outside of expected ranges (for example, reported at 0 or greater than 200).
In fact, when we removed the chest strap data that was clearly erroneous from our analysis we obtained results that were 98% accurate:
So what does all this mean?
Nothing is perfect. When looking at the accuracy of heart rate monitoring devices, remember that even the “gold standard” has errors. When you are comparing any device measuring heart rate to a chest strap, 92-94% of the data within +/-5% is the BEST you can expect from that device during activity. If you see results that are higher than that, just understand that the evaluators are very likely not including data where the chest strap is not reading correctly. Remember, if the data looks too good to be true, it probably is.
If you are interested in seeing the details behind this research in a white paper, reach out at info@valencell.com and we’d be happy to send you a copy.
Hello,
who ever stated chest straps are the golden standard? Where is the original source?
I can magine multilead ECG with beat to beat corrected HRV will be more accurate as standard reference.
Please send me a copy of the white paper,
feel free to contact,
best,
Johan
First, chest straps are more accurate than ANY (including your ear-based) HR monitors and certainly for HRV. Your experiment was flawed as your first image above illustrates. These chest straps are designed to be positioned directly below the pectoralis muscles and one of those cannot be positioned correctly. You should have switched the position of the two straps and see if you could repeat the results. Now, chest straps are simply the most accurate “consumer” heart rate devices. For example, when the Cleveland Clinic studied wrist-based HRMs and a chest-based HRM, they used a clinical ECG stress system for comparison. I would love to see how your ear-based devices compare to the real gold standard (we can actually measure the beat to beat HR from the stress ECG)?
Dr. Albert,
The intent of the post and experiment was to provide some perspective on the perception in the market that ECG chest straps are perfectly accurate. The point is that ECG chest straps, just like all biometric measurement tools, have errors and make mistakes.
Regarding the experiment, we switched the position of the two straps as you referenced. Here is some more detail on the methodology:
Trials examining validity of the BLECS (polar H7, Polar Electro, Kempele, Finland) were completed using twenty-three healthy subjects (characteristics in table 1). Each participant completed a single, dynamic, 8-minute treadmill session consisting of standing, walking and running in order to elicit a variety of exercise intensities (protocol details in table 2). Subjects wore two chest strap heart rate monitors with random placement in regards superior/inferior placement (figure 1): one utilizing wireless communication to a watch (CSHRM) (polar cx800, Polar Electro, Kempele, Finland); the second BLECS transmitting to a smart phone fitness application capable of exporting heart rate data for analysis. Heart rate was recorded continuously from both devices and statistical analysis was completed using 1-second data reporting. Data was aligned in post processing to achieve maximal correlation. This alignment is necessary as data collection can be offset by several possible factors including, but not limited to: data being collected onto separate devices, clock differences between devices, and latency. Analysis also examined accuracy related to heart rate intensity (< moderate and ≥ vigorous). Intensities were identified utilizing the American College of Sports Medicine criteria for moderate (45-65% of heart rate reserve) and vigorous intensities (65-89% of heart rate reserve) (ACSM, 2011). Statistical analysis included mean heart rate, bias as percent error, mean absolute percent error (MAPE), and frequency distribution within ±5% windows.