Valencell recently completed testing of 10 different optical heart rate monitors for wrists and arms that included sport watches, smart watches, fitness bands and Valencell reference designs. The full report continues below and you can also download the report here.

 

PERFORMANCE OF LEADING OPTICAL HEART RATE MONITORS DURING INTERVAL EXERCISE CONDITIONS

Valencell Biometrics Lab

January 2017

Vman_pms_orange

Why we test

As a developer of optical heart rate monitor (OHRM) technology, Valencell has a critical need to objectively understand how all OHRM technologies in the market, not just Valencell’s, perform in all conditions. We want to know how these technologies perform, why they perform the way they do, how they work, and under what conditions they don’t work. If our technology is not performing to our customers’ high standards for accuracy, our business will suffer. Period.

As you will see from the results in this report, Valencell technology is performing at the highest levels in the industry.

Valencell Biometrics Lab

We created the Valencell Biometrics Lab to serve not only as a testing ground for the latest in biometric wearable technology, but as a check & balance on our product development teams to ensure the highest standards. The lab is run by Dr. Chris Eschbach, PhD exercise physiologist, and operated by a team of exercise scientists who have tested thousands of devices of all types on thousands of test subjects over the years.

Valencell Biometrics Lab stats - Jan2017

The Lab maintains a rotating pool of nearly 100 volunteers who visit the lab every week to participate in testing. The volunteer pool consists of a broad range of ages, weights, fitness levels, skin tones, and physiological habitus, which enables Valencell to test how devices work across a diverse population of users.

Testing methodology

This particular investigation involved testing 9 different devices and two of those devices in multiple positions on the arm. Subjects included 30 healthy adults (15 males and 15 females), ranging in age from 21 to 68. At most four devices under test were worn by the subject at one time – one on each wrist and arm (forearm or upper arm), respectively. All data was compared to the Polar H7 BLE chest strap heart rate monitor as a benchmark. Devices were always worn according to the manufacturer’s directions, and there was never more than one wrist device worn on the same wrist. All subjects performed the same test with all devices: an 8-minute indoor dynamic treadmill test, commonly known as the “Valencell Test”:

Valencell 8-min test protocol

Over years of testing, the Valencell Test has been shown to identify weaknesses in OHRM devices, particularly in tracking changes in heart rate and other biometrics caused by varying activity intensity in different intervals.

Devices tested

The following devices were tested:

  • Apple Watch
  • Garmin Fenix 3 HR
  • Garmin Forerunner 235
  • Fitbit Surge
  • Mio Alpha 2
  • Motorola 360
  • Scosche Rhythm+ at the forearm
  • Scosche Rhythm+ at the upper arm
  • Suunto Spartan Sport Wrist HR
  • TomTom Spark
  • Valencell reference design at the forearm
  • Valencell reference design at the upper arm
  • Valencell reference design at the wrist

Device Placement, Recording & Data Extraction

Wrist Devices

Subjects wore one wrist device on each arm during testing, and the devices were assigned at random to the subject’s dominant and non-dominant sides. All devices were put on by the tester to ensure consistency. Placement was approximately one finger proximal to the ulnar styloid process. The device was tightened to the point where it would not slide when the subject shook their wrist but was not uncomfortable or squeezing the arm.

Forearm Devices

Subjects wore one forearm device on each arm during testing, and the devices were assigned at random to the subject’s dominant and non-dominant sides. All devices were put on by the tester to ensure consistency. Placement was approximately two fingers distal to the elbow crease and aligned with the radial artery. The straps were tightened snuggly to prevent sliding.

Upper Arm Devices

Subjects wore only one upper arm device at a time because they wore an armband with an iPod Touch on the other arm. Devices were assigned at random to the subject’s dominant and non-dominant sides. All devices were put on by the tester to ensure consistency. Placement was approximately four fingers distal to the shoulder joint on the lateral side of the arm, directly on top of the deltoid muscle.  The straps were tightened snuggly to prevent sliding.

Data Collection by Device

  • Apple Watch: Data was collected on the watch using the iOS Health App. The watch synced with and iTouch and the heart rate data was download as a CSV file. The data is recorded every five seconds, so every fifth row was extracted. Five-second averages were then taken from the chest strap heart rate data.
  • Fitbit Surge: The test were recorded by the device’s “Outdoor Run” option and then synced with the Fitbit app. The tester connected the watch to GPS, even for the indoor test, in order to be able to download the data. Once the device was synced, the data could be exported to a TCX file from the Fitbit website.
  • Garmin Fenix 3 HR and Forerunner 235: The tests were recorded by the device’s “Run” option. The watch was then synced with the Garmin Connect app and the data was exported as a TCX file. Both devices had to be switch to one-second recording as opposed to the pre-set “smart recording” mode.
  • Valencell reference design (wrist, forearm and upper arm), Scosche Rhythm (forearm and upper arm), Mio Alpha 2: Data from these devices was recorded through BLE on Valencell’s iOS data collector on an Apple iTouch. Data is recorded every second. Subjects wore the phone on an arm band during the tests.
  • Motorola 360: The device was paired with an Android phone with the MotoWear and MotoBody app. MotoWear was then connect with Strava, so all workouts would automatically sync to Strava as well. This was done because there was no option in the Motorola apps to download the data. Strava, like Fitbit, only allows extraction of data when there are GPS records. Therefore, the Motorola 360 tests were all recorded in “Outdoor Run” mode and the tester had to ensure GPS was found.
  • Polar A360: The test were recorded by the device’s “Indoor Run” option and then synced with Polar WebSync. Data could then be downloaded directly to a CSV file.
  • TomTom Spark: The test was recorded by the device’s “Treadmill Run” option and then synced with the TomTom MySports app. Data could then be downloaded directly to a CSV file.
  • Suunto Spartan Sport Wrist HR – The device was paired to a computer by Bluetooth for data collection with a Valencell-built, multi-purpose data collection utility application. The software produces a CSV file of results for subsequent analysis.

Sources of Error

  • Sweat occasionally caused the devices to slide, even when the test started with the proper tightness and fit.
  • Due to arm and/or wrist circumference, the devices were in a slightly different location and tightness on some participants.
  • The Fitbit Surge was a small band, so it was tighter and closer to the wrist than ideal.
  • The Garmin Fenix watch face was too large for some wrists and the band could not be tightened enough to prevent sliding.
  • The Valencell Reference Design at the upper arm had to be lower on the arm for several participants.

It is also important to note that it is common for the benchmark chest straps to experience errors. Valencell has done simultaneous testing with multiple CSHRM devices and found that they typically produce results within +/-5% of each other anywhere from 95-98% of the time.

Test Results

Valencell Test results - Jan2017

*Valencell-powered product

Conclusions

While many different products on the market today claim to monitor heart rate at the wrist, there remain significant differences in the accuracy in these OHRM products. This research clearly shows the accuracy in these devices varies significantly, particularly during interval training that involves numerous changes to the heart rate in short periods of time.

Valencell has dedicated significant R&D resources to solving many of the challenges with measure biometrics at the wrist and devices powered by Valencell technology continue to outperform.