A recent validation study was published by Trowell et al. (2024) comparing an early version of the Metric VBT app (v0.6.0) with 3D motion capture.
While Taber et al. (2023) demonstrated that the Metric beta (v0.5.4) was valid and reliable when compared to 3D motion capture for measuring barbell velocity and range of motion in the squat, bench press, and deadlift, this newly published study from Trowell et al. did not make the same recommendation.
While the results weren't as strong as we'd hoped, in the interests of transparency and ongoing development, in this blog post I share the results from this paper along with our improvements made since Metric v0.6.
Check out the paper and download the full text PDF at this link →
Study details
Lead author: Danielle Trowell
Title: Validation of a commercially available mobile application for velocity-based resistance training
Metric version tested: v0.6.0 (beta)
TL;DR: Metric VBT (v0.6.0) showed mixed results when compared to 3D motion capture, with accurate squat rep detection but limited accuracy in velocity and rnage of motion measurements for both squat and bench press, indicating the need for improvement in velocity-based training applications.
Study methods
The study compared Metric VBT (v0.6.0) to a Vicon 3D motion analysis system for accurately counting repetitions, measuring range of motion (ROM) and mean velocity on squat and bench press.
This study pushed Metric to its absolute limits, looking at 4000+ repetition samples for the squat and bench press through the use of 24 subjects and up to 5 device placements per repetition for comparing Metric data across a range of recording angles (+20º → -20º). The study used multiple devices both older and newer models (iPhone 8 - iPhone 13 Pro).
- 24 healthy individuals (15 males, 9 females) with varying resistance training experience participated in the study.
- Back squat and bench press loads from 25-100% of 1RM were used
- Measurement Devices:
- Metric VBT app (v0.6.0) running on various iOS devices
- Vicon 3D motion analysis system (gold standard)
- Camera Positions: Five angles (-20°, -10°, 0°, +10°, +20°) relative to the direct end of the barbell
- Measured Variables:
- Repetition detection
- Barbell range of motion (RoM)
- Mean barbell velocity
Study Results and Key Statistics
Metric was not found to be a reliable velocity tracking tool in this study.
We'll explain why this was not a surprise to us here at Metric later in the article but first, let's break down the key findings from the study.
Understanding CCC Scores:
The study uses Lin's Concordance Correlation Coefficient (CCC) to measure agreement between Metric and the Vicon system. CCC scores range from -1 to 1, where 1 indicates perfect agreement. Generally, scores are interpreted as:
- ≥0.99: Almost perfect agreement/correlation
- 0.95 - 0.99: Substantial (high) agreement
- 0.90 - 0.95: Moderate agreement
- < 0.90: Poor agreement
Now, let's look at the results:
1. Metric was very good at counting reps, especially for squats, but had some issues with bench presses.
- Squat: Excellent performance with >99% precision and recall across all camera angles. Only 5 missed repetitions out of 2,195 performed, with zero false positive (ghost) repetitions.
- Bench Press: Lower accuracy with precision ranging from 94.0% to 97.3% and recall from 97.9% to 98.9%. This means Metric sometimes missed reps or counted extra ones that didn't happen.
We believe the bench press issues were due to the shorter movement range, the curved bar path, and ghost reps from racking/unracking the bar between sets.
2. Metric was reasonably accurate in measuring Range of Motion (ROM), but had room for improvement.
- Squat was again tracked better than bench press
- Moderate to substantial agreement with Vicon (CCC: 0.941-0.975).
- Average Limits of Agreement (LoA): -5.45 to 4.94 cm.
- Bench Press had a lower correlation with 3D capture, but still some agreement
- Moderate agreement with Vicon (CCC: 0.901-0.926).
- Average LoA: -5.80 to 3.55 cm.
Put simply, ROM tracking with Metric v0.6 were off by ±5 cm (2 inches) on average for both bench press and squats. a bias of 5cm might be tolerable for some general tracking purposes but the fact that Metric provided ROM results both +5 and -5cm from the MOCAP baseline leaves the Metric beta v0.6 with limited usefulness.
3. Metric struggled with Mean velocity validity even more. It did show glimmers of reliability, consistently overestimating bar speed vs 3D motion capture.
- Squat:
- Poor agreement with Vicon (CCC: 0.826-0.890).
- Consistent overestimation (mean bias: 0.09-0.13 m/s).
- Average LoA: -0.03 to 0.25 m/s.
- Bench Press:
- Poor to moderate agreement with Vicon (CCC: 0.881-0.931).
- Consistent overestimation (mean bias: 0.06-0.09 m/s).
- Average LoA: -0.08 to 0.26 m/s.
Metric's mean velocity measurements were consistently faster than MOCAP by 0.1-0.13 m/s on the squats and 0.06-0.09 m/s on the bench press
While Metric has a consistent bias (something that is useful from a reliability standpoint, these discrepancies weretoo wide for Metric v0.6 to be a valid velocity tracking tool, indicating a need for improvement in this area.
4. Metric v0.6 became less accurate the greater the camera angle from the end of the bar
- Generally, accuracy decreased as the camera moved further from perpendicular (0°) to the participant.
- The 0° position typically showed the best results across measurements, while ±20º was the worst.
In summary, while this early version of Metric (v0.6.0) was able at counting squat reps and was able to “Ballpart on both ROM and mean velocity, it needed significant improvement to become a professional-grade tracking tool.
Improvements to the Metric tracking algorithm since beta
Thankfully no one is using the 0.6.0 beta version of Metric in their training anymore!
v0.6.0 signalled the final publicly available beta version of Metric, eventually superseeded with the major release of Metric v1.0 in November 2022.
Version 1.0 Milestone
Shortly after data collection ended in this study, we launched Metric version 1.0, featuring two key new features in how we detect and track barbell lifts.
- Intelligent rep classification. Determining the rep start and end points is surprisingly difficult. With v1.0 we rewrote the Metric smart algorithm to find these key moments in your reps for a significant boost to accuracy. This system also enabled handling for ballistic movements like snatch and clean variations.
- X & Z axis correction. X-axis and Z-axis correction allows for better correction and sensitivity to both curved bar paths, and the specific device placement and recording angle used in each set. This aligns with the corrections used by other velocity tracking technologies such as Gymaware and RepOne that also offer axis correction for improved accuracy.
Ongoing iterative improvements
These findings have been crucial in guiding our development efforts since Metric left its beta stage.
v1.0 showed significant improvements in reliability and validity, affirming that computer vision is capable of delivering high precision training data. To accelerate development further, we opened up a public bug reporting feature directly in the Metric app, allowing lifters and coaches to send us their videos and tracking data fro inspection.
Thanks to this community feedback we made huge strides between v1.0 and today making Metric faster, more efficient, and most importantly more accurate in the delivery of training data.
Some of our key areas of tracking improvement since v1.0 have been
- Camera zoom and plate focus lock to accomodate more phone placements
- Reliability for a wider range of phone positions, including placing the phone lower down relative to the barbell. Metric is now reliable between +25º and - 25º from the end of the bar.
- Advanced camera mode for reducing motion blur on explosive lifts
- Low light detections and tracking are handled better with more real-time video processing for sharper image detection
- Handling for longer in-set rest periods to allow for a single video to record and deliver results for an entire set of clusters.
- Faster video processing for reduced processing time and device battery efficiency
- Numerous edge case filtering and corrections
Thank you for your continued support and the ongoing bug reports and feedback, they help us to continue making Metric better and inform where we focus our energy!
Stay tuned for more updates, and as always, we welcome your questions and insights.
You can download the Metric VBT app for iOS at this link →
UPDATE: Positive Validation for Metric v4.5.0
This hard work has paid off with a recent validation from masters student Antonio Šagovac who found that Metric VBT version 4.5 showed strong correlations with the Vitruve linear position transducer on the bench press. This study is the first to validate a post-beta version of Metric VBT, showing the robustness and accuracy of the app in real-world conditions.
