Last week the first ever validation paper for Metric VBT was published in the International Journal of Strength and Conditioning.
The research was done with an early version of Metric, and the result was outstanding, demonstrating that Metric VBT is a valid and reliable solution for coaches.
This comes as no surprise to us based on our previous in-house testing.
Chris Taber and his team at Sacred Heart University in Connecticut have a done a very thorough job in their data collection for this paper, completing over 800 reps comparing the results from Metric VBT and a 3D motion capture system. I strongly encourage you to check out the paper for free here.
Study details
Lead author: Chris Taber
Title: Validity and Reliability of a Computer Vision System to Determine Bar Displacement and Velocity
Date of publication: November, 2023
Metric version: v0.5.4
TL;DR: This investigation revealed that VBA [Velocity-based Application, Metric] is a valid and reliable measurement tool in the measurement of MCV and ROM. This new VBA provides coaches with a low-cost, simple, phone-based application which can be used for VBT.
Validation for the Metric beta v0.5.4
This study used a beta version (v0.5.4) of the Metric app back in July 2022 for their data collection. So while the results of this validation are strong, we have made many strides to improve further. As of November 2023, The Metric app is now up to version 3.2.2, so as you can imagine a lot has changed and improved in the past 15+ months!
That does not invalidate this research in any way; the break-through technique behind our proprietary computer vision algorithm was invented in 2021, and remains the foundation of Metric today. But users of the curent app version now have access to even better data precision than what was available when this study was conducted.
Since the data collection for this study, Metric has been improved with so many incredible technical achievements:
- Adjusted the algorithm to accurately record data from a wider range of angles with X & Z axis correction
- Invented a unique rep classifier model that dynamically establishes rep start-end points
- Added support for ballistic lifts with reduced motion blur
- Refactored the algorithm code to massively improve processing (up to 400% faster in many cases) and improve reliability between devices and phone positions
- Created an in-set calculation method to enable our brand new real-time feedback feature
(Big shoutout to our Lead Engineer David for this insane list of accomplishments!)
Validity, reliability, and the “gold standard”
Here at Metric we consider a calibrated 3D motion capture system in lab settings the “gold standard” for recording velocity. Unfortunately a 3D motion capture setup like this costs tens of thousands of dollars, and isn’t exactly portable or easy to use in most gym settings!
When measuring Metric we care about two things: validity, and reliability.
Validity means the data is very closely matching reality. Eg. if a 3D motion capture system says a rep was 0.98m/s, and Metric shows 0.99m/s, that would be considered highly valid.
Reliability means the data is consistent; that it is doing the same thing repeatedly across reps, sets and even over different days and environments.
A technology has more room to be imperfect on the measure of validity, but as long as it is consistently reliable it will be a useful training tool and can guide decision making in programs.
Our goal has always been to build a tool that anyone can download to their phone, and within minutes record velocity data with results that are valid enough compared to a 3D motion capture, but with incredible reliability across environments and contexts so that they can rely on Metric for training decisions.
Reliability and validity results of Metric VBT
In summary, Metric (and remember this was v0.5.4) performed really well on when compared to 3D motion capture.
On all three exercises (Bench Press, Back Squat, Deadlift), across both fast and slow sets, Metric was able to provide mean velocity and range of motion data that was both valid and reliable.
Significant correlations were found between VBA and MC for both ROM and MVC on slow and fast repetitions.
Overall Metric showed very little bias in its reported velocity, for most conditions only 0.01 - 0.02m/s - fast squats showed a larger bias of +0.06m/s - something we have since calibrated Metric to account for, while bias in range of motion was incredibly tight - under 2cm (0.8 inches) for all exercises and speeds.
- Displacement (ROM) showed correlations ranging from moderate to nearly perfect, with an r value of 0.43-0.94.
- Mean velocity correlations were even better ranging from good to nearly perfect with r values ranging from 0.67-0.95.
- Displacement (ROM) achieved an intraclass correlation coefficient (ICC) of 0.67-0.98.
- Mean velocity ICC were even better between 0.79 - 0.98.
Practical take-aways
While the study's findings are positive, we're in the business of authenticity, not perfection.
To be up-front, it is actually impossible for Metric (or any other velocity based training device or app) to perfectly match motion capture, because each lab setup has a fixed determination of what qualifies as the start and end of each rep set by the researcher. This will always differ from Metric.
In fact the paper discussed this:
…the results could have been affected by differences in analysis choices between our custom software and Metric VBT, such as filter design and cutoff frequency and the method for identifying the concentric phase.
You would think that the start and end of a rep is simple to quantify… but back when we were doing our very intial R&D for Metric we were surprised to discovered that it is not! Every VBT technology has a different approach to this problem, and thus they all show different results in real world use.
We are particularly proud of our unique rep classifier model launched in version 1.0.1 which massively improves how Metric determines the start-and-end points for all reps. This version was released not long after this study ran its data collection. Hopefully some validation on this model will become available in the near future.
So the real take-away from this paper is that Metric is a valid and reliable tool for velocity based training. With all the incredible technical updates rolled into Metric since version 0.5.4, you can confidently rely on Metric to help you coach your athletes or make better training choices.
