Beating heart surgery is very difficult to perform because the surgeon has to work on the heart while its surface is rapidly moving. A remote-controlled robot can be employed to automatically compensate for the motion of the beating heart. By showing a stabilized image to the surgeon, the illusion of operating on a still heart can be created.
The research on this problem was carried out in cooperation with the Heidelberg University Hospital as project T2 of the Research Training Group 1126 on “Intelligent Surgery” (GRK 1126). An article on my work has appeared in the APR Newsletter 2015.
The goal of image stabilization consist in creating a video stream of the beating heart where the movement of the heart has been removed. However, changes to texture as caused by manipulation by the surgeon are supposed to be retained. Thus, we want to combine the shape of the heart from a certain reference frame with the current texture information.
I have published several papers on this problem, which focus on different issues.
This video illustrates some of my results.
In literature, most methods for beating heart tracking rely on cameras exclusively. Occasionally, other sensors, such as ECG or a blood pressure sensor are used. A key to achieving accurate, reliable, and robust tracking is the fusion of information from multiple sensors that complement each other and that can compensate for each others weaknesses.
Motivated by the availability of novel sensors, I investigated the applicability of small inertial measurement units (IMUs) contained in a single chip to the problem of beating heart tracking. Furthermore, I performed experiments with a Microsoft Kinect camera based on the structured light measurement principle. Both types of sensors showed promising results .
For the purpose of beating heart tracking as well as three-dimensional image stabilization, an accurate 3D reconstruction of the heart surface is essential. However, this is a fairly tricky problem due to the constant movement and elastic deformation of the heart surface. Thus, classical algorithms that assume a rigid surface are not applicable. Furthermore, it is desirable to combine information from color cameras (which is usually fairly accurate, but not very dense, and can be affected by specular reflections) and information from depth cameras (which tends to be less accurate, but more dense and more robust with respect to specular reflections). For this reason, I proposed a novel algorithm on recursive surface tracking using a combination of depth and color information , .
The idea of heart phase estimation is to condense the state of a complex dynamic system such as the beating heart into a single quantity, the phase. The intuitive meaning of the phase is the answer to the question how much of the current heart beat has already passed. Because phase is a periodic quantity, I employ an estimator based on directional statistics .
In order to simplify tracking of the heart surface, we place artificial landmarks on the beating heart. These landmarks can easily be located using computer vision algorithms due to their distinct color. However, all artificial landmarks look identical, so they cannot be distinguished. This creates an assignment problem as it is common in the field of multiple target tracking.
To address this problem, I devised a multiple target tracking method using an algorithm based on the Kernel-SME idea originally proposed by Marcus Baum .