Bernstein member involved: Ilka Diester

A team led by Artur Schneider and Prof. Dr. Ilka Diester from the Institute of Biology III and Prof. Dr. Thomas Brox from the Department of Computer Science at the University of Freiburg have developed a new system that records the 3D movement of freely nameable body points: “FreiPose” uses several cameras and a special network architecture to recognize defined body points and track them. The method enables researchers to exclude movement influences of certain body parts in their analysis that are not of interest. The group presents their new method in the journal Neuron.

Body points are reconstructed directly in 3D using different camera perspectives and suitable network architecture

“We asked ourselves whether it is possible to link neuronal activity in the brain to the movements of individual body parts when living beings move freely and only perform spontaneous movements,” Diester explains. According to the researchers, two conditions must be met for this to happen: First, detailed 3D tracking of movements at the level of individual body parts is needed; second, the body part of interest must be isolated from the influence of other body parts.

The scientists have created a system with “FreiPose” that reconstructs detailed postures and movements of individual body points directly in 3D using various synchronized camera perspectives. The body points are freely selected beforehand.

“The novelty of FreiPose is the projection of learned image features from individual camera views into a common 3D space, where a network architecture is able to combine all information to draw conclusions about the position of body points. Due to its multi-view approach and native 3D reconstruction, FreiPose is particularly suitable for free-moving creatures in various environments, including in those with obstacles,” says Brox.

“FreiPose” could be used for epilepsy patients

To see the neuronal representations of movements of individual body parts, researchers reduced the contributions of each other’s body parts and movements. “This strategy makes it possible to analyze the behavior in question, for example, only hand movements. The method could be used, for example, with epilepsy patients with implanted electrodes. If neuronal activity is measured to check the electrodes, FreiPose could be used to track the patient’s movements and factor out the influence of unwanted movement,” Diester explains.

“This method can be used to better study the behavior of living beings because they can move completely freely and naturally,” adds Brox. “Once established, our system can be applied to large data sets, thereby minimizing workload.”