Stereoscopic liver surface reconstruction

Introduction

Nowadays one of the most important directions in radiology and surgery is computer-aided diagnosis and therapy [1]. As regards computer-aided surgery, the research is focused on image-guided procedures. In order to construct the navigation system, information on movability is required, especially as regards the soft tissue in the abdomen.

Our objective was to develop a non-invasive methodology of liver surface points detection and tracking. There are many approaches relating to computer-assisted surgery, e.g. laparoscopic teleoperated robot-assisted surgery, da Vinci, image-guided laparoscopic surgery, and laparoscopic surgery computer simulators [2, 3]. Our approach was to use standard equipment (laparoscopic cameras, typical surgical tools), and not interfere with medical procedures. We also planned to use our approach for both laparoscopic and single incision laparoscopic surgery [4, 5].

While developing the methodology we took into account existing approaches. Many works have been done in beating heart surgery. Bogatyrenko et al. proposed a stochastic physical model of the beating heart surface [6]. Acquisition was done with a trinocular camera system [7]. Cortadellas et al. proposed the concept and overview of an augmented reality system that assists laparoscopic surgical procedures [8]. Mountney et al. proposed stereo cameras, a complex anatomical structure movement based on a modified Bouguet algorithm and used phantom tissue to model 3D geometry [9]. There also exists heart motion estimation and prediction using a Fourier series model, a vector autoregressive model and the extended Kalman filter. Groeger et al. proposed a laparoscopic tool tracking colour base segmentation and solved the problem of deformation and illumination occlusion of objects [10]. Chen et al. presented a feasibility system for guiding the robotic removal of the prostate that will provide augmented reality surgical guidance [11]. A stereoscopic fibroscope imaging system for minimally invasive surgery (MIS) is presented in [12]. The major goal was to design flexible instruments that can follow curved anatomical pathways by using the Simultaneous Localisation and Mapping (SLAM) algorithm.

An overview of the literature shows that the data about liver motion for a specific patient are not known so our future development work is regarded by us as a milestone. In the results of our work we want to use in real time a non-rigid registration algorithm of liver motion. This method focuses on simple algorithms which can be used in deterministic short time and are applicable to an online processing approach with a constant constrained frame rate in consultation with the surgeon.

Aim

The paper presents a practical approach to measurement of liver motion, both respiratory and laparoscopic, with a tool guided in the operating room.

Material and methods

Our approach can be divided into four steps: equipment setup, camera calibration, selecting the 3D reconstruction algorithm, selecting the triangulation points.



Equipment setup



The first and most important principle of our approach was to develop a non-invasive methodology (Figure 1). Laparoscopic cameras are nowadays standard equipment in the operating room, so we have focused on computer vision algorithms and we need at least two cameras to obtain a pair of stereoscopic images. However, there are some monocular reconstruction algorithms but there are many limitations which are difficult to take into account during in vivo laparoscopic reconstruction, e.g. requirement of a preoperative 3D model and ambiguity of depth map reconstruction. In order to apply a computer vision paradigm we need to mount the cameras together rigidly, so we prepared a special connecting device as seen in Figure 2. To set the system up we use two complete laparoscopic vision systems: Olympus CV-180 video processor HDTV, CLV-180 Light Source and an “Endo-Eye” laparoscopic camera with a Matrox Orion HD frame grabber. One important part of the setup is to synchronize the video stream. Generally trigger input is not included in the laparoscopic video acquisition system. This was the reason we needed to develop a software camera synchronization solution. This is based on an application with two background threads, and to balance them to record equal frames on both cameras. We use a global counter in the main thread to mark the frames. For laparoscopic surgery we use an Olympus LESS trocar.



Camera calibration



In our research we tested two calibration methods, the Tsai algorithm [13] and the Bouguet [14] method, both with satisfactory results. The Bouguet algorithm is based on a simplified pinhole camera model and low computation requirements. The Tsai method is more complex and can be divided into two steps: calculating the pose (3D orientation, and x-axis and y-axis translation) in the first stage, and computing the focal length, distortion coefficients and the z-axis translation in the second stage. We use the calibration grid and OPENCV Library (Hough Transform) to get sub-pixel coordinates of corresponding points and calibrate the camera with the Roger Y. Tsai algorithm [15]. For calibration we used two perpendicular calibration grids (chessboards). We gained the best results when the number of calibration points was greater than 30.



Selecting the 3D reconstruction algorithm



There are many approaches to the 3D reconstruction problem, but there are many limitations to its use in the laparoscopic reconstruction. Difficult cases with shape from shading (SfS) need multiple light sources [16, 17]. Because of liver surface reflection we use only one light source and the SfS is not practical for our approach. The texture of the liver surface is not distinct, so we also reject the shape from texture (SfT) [18] approach. After evaluation of different reconstruction approaches, we decided to use the classical triangulation technique. Despite the imperfections of the triangulation method, such as sensitivity to illumination and artefacts that cause pixel occlusions, we decided to adopt this method and prove its utility, and obtained high accuracy. Our approach is more challenging than one by attaching landmarks or colour markers to the liver surface. Triangulation t is based on finding the position of a point in 3D with known camera matrices and projections of a point into both views (Figure 3). There are two types of estimation assessment: root-mean-square (RMS) residual error (distance of the measured and the estimated value) and RMS estimation error (distance of the estimated and the true value) [19]. We focused on open surgery, but our approach was flexible and could be used in laparoscopic surgery. Two rigid laparoscopic cameras were used.



Selecting the triangulation points



As regards state of the art 3D reconstruction algorithms, the manual finding stereo correspondence seems not to be the perfect approach, but because of nonlinear shape changing in different segments of the liver, we decided to use it. Our main motivation was to verify the general knowledge of liver motion and collect specific knowledge to propose the proper shape deformation constraints to deploy the monocular shape from a motion approach (SfM) to the liver surface (based on standard laparoscopic equipment) [20]. Our preliminary approach was to use well-known solutions, i.e. RANSAC and Lucas-Kanade [21]. We tested Image and Video Processing – MATLAB and we were not satisfied with the RANSAC results. Secondly we evaluated the Lucas Kanade method in OPENCV for the optical flow of the liver. The reasons for the failure of both methods were lack of corners and edges, and a modest gradient on the limited visible part of the surface.

The fundamental problem of using the RANSAC algorithm and the other method (optical flow) for liver images is that the liver is not a rigid body. It is difficult to have a basis for the results obtained automatically for three reasons:

• ruffles on the surfaces (some points of the surfaces disappeared in the frame sequence),

• specific lightning caused by abdominal characteristics and the construction of the laparoscopic camera (after the pilot study we used only one light source to minimize the reflection of light effect),

• liver deformation (the distance between specific corresponding points in the frame sequence is not constant).

Our data set consisted of 10 stereo video sequences for open liver surgery procedures. We also recorded 2 laparoscopic procedures using the SILS technique. After obtaining the videos from the cameras we tried to select corresponding frames using the global counter (Figure 4). During the video acquisition we used two balancing threads to grab frames from two cameras and mark frames with the value of the global counter. The steps of preparing data for the triangulation algorithm were:

• select the corresponding frames based on the global counter values,

• mark the characteristic points.

In order to assess the intra-operator variability in points of interest (POI), the manual clicking of the POI positions on selected stereo images was repeated twice by two operators. We tracked from five to

ten POIs on each frame which were localized on specific liver segments. Basically we tried to use anatomical landmarks, but it was difficult, or even impossible. Therefore we decided to choose local extremes (Figure 5).

In practice, we used imtool and imshow standard MATLAB image analysis tools (Figure 6). We also tried to use colour base segmentation. The motivation was that the colour of standard laparoscopic tools (silver or black) is different from the liver surface colour (hue of brown). Using this approach it was possible to find the contours of the laparoscopic tool, but impossible to find specific points. Because of our principle of using specific marked points, we discarded this approach.

The proposed evaluation method based on both calibration points of the chessboard reconstruction and measurements made by the Polaris Vicra tracking system are used as a reference system.

Results

We have presented separate examples for left and right liver lobes (Tables I-II). These results were obtained by performing the triangulation algorithm on manually found points in corresponding video frames. The first six left columns present the two different position of the same POI in a world coordinate system (measured in mm) in video sequence, the next two RMS residual error of reconstructed positions, and the last three the displacement vector. We obtained on average 3-4 mm RMS residual error with manually selected corresponding POIs. Due to vanishing pixels in the video sequence, RMS residual error of some POIs is up to 10 mm.

Discussion

Generally the results confirm the expectations of the scope of liver movement [22, 23]. Respiratory motion is one rank lower than the possible motion guided by the laparoscopic tool. It is possible to reconstruct liver motion using the SILS trocar but it is more complicated to set up the equipment using two single view cameras than with open surgery. Using one stereo camera (e.g. da Vinci Surgical System) will be a great simplification as regards camera calibration and mounting the camera in the SILS trocar, but this is not commonly accessible in operating rooms nowadays.

Finding the correspondence points manually requires deep image analysis, which allows observation of the wrinkles of the surface of the liver. Some points of the surface disappear in the frames sequence. It creates a challenge to use monocular SfM reconstruction even to find the correspondence points automatically.

The presented pilot study triangulation approach to liver surface tracking with manually selected correspondence based on two standard monocular laparoscopic cameras needs improved accuracy to allow medical application, e.g. tracking respiratory motion during radiotherapy. We are planning our future work in two directions. One is to find correspondence points automatically with sub-pixel accuracy. Therefore we will try to use a specific Olympus camera narrow band imaging which is the camera specific mode to enhance the vessels to aid finding the POIs. The second is to try to use only one camera for monocular surface reconstruction.

Our pilot study is a preliminary stage of development of a navigation system for treatment of unresectable liver tumour. The navigation system allows the surgeon to track the current position of surgical tools as regards the positions of anatomical structures during intervention; therefore tracking movement of anatomical structures is required. There are a few methodologies to find a solution. Bogatyrenko et al. [6] proposed a stochastic physical model for the similar problem of beating heart surface tracking. Mountney et al. developed stereo cameras, a complex anatomical structure movement based on a modified Bouguet algorithm and used phantom tissue to model 3D geometry [9]. Groeger et al. used a laparoscopic tool tracking colour base segmentation and solved the problem of deformation and illumination occlusion of objects [10]. All of them need special equipment to obtain positive results: stereo camera [9], colour laparoscopic tool [10] or complex mathematical models [6], that need to be adjusted to a specific patient. We have proposed a methodology based on standard equipment and does not require complex mathematical calculations. A disadvantage of our method is extracting correspondence POIs manually, but it could be overcome by using specific POI image descriptors or applying the monocular shape from motion algorithms.

Acknowledgments

The authors would like to acknowledge the financial support from the Polish National Centre for Research and Development (NCBiR). Grant no. LIDER/ 03/47/L-1/NCBiR/2010 “Development of planning system and computer aided minimally invasive surgery for hepatic and metastatic liver carcinoma localization, diagnosis and destruction”. Open surgery procedures were registered in the Department of Minimally Invasive Surgery and Proctology at Wroclaw Medical University, SILS laparoscopic procedures in the Chair and Department of General, Transplant and Liver Surgery at the Medical University of Warsaw. Our work is the preliminary research in this grant funded project.

References

 1. Peters T, Cleary K. Image-guided interventions: technology and applications. Springer, London 2008.

 2. Budziński R, Michalik M, Frask A. Education in laparoscopic surgery. Videosurgery Miniinv 2008; 3: 22-9.

 3. Gruca Z, Kobiela J, Stefaniak T. Surgical simulator in didactics of minimally invasive techniques. Videosurgery Miniinv 2008; 3: 30-4.

 4. Kurpiewski W, Pesta W, Kowalczyk M, et al. SILS cholecystectomy – our first experiences. Videosurgery Miniinv 2009; 4: 91-4.

 5. Łosin M, Czauderna P, Gołębiewski A. Single incision laparoscopic surgery cholecystectomy in children – preliminary experience. Videosurgery Miniinv 2010; 5: 139-41.

 6. Bogatyrenko E, Pompey P, Hanebeck U. Efficient physics-based tracking of heart surface motion for beating heart surgery robotic systems. Int J Comput Assist Radiol Surg 2011; 6: 387-99.

 7. Richa R, Bo A, Poignet P. Beating heart motion prediction for robust visual tracking. IEEE International Conference on Robotics and Automation 2010; 4579-84.

 8. Cortadellas J, Bellaire G, Graschew G. New concepts for intraoperative navigation: calibration of a 3-d laparoscope. Bildverarbeitung fűr die Medizin, Springer, Berlin 2000; 158-62.

 9. Mountney P, Stoyanov D, Davison A, et al. Simultaneous stereoscope localization and soft-tissue mapping for minimal invasive surgery. MICCAI'06 Proceedings of the 9th International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, Berlin 2006; 347-54.

10. Groeger M, Arbter K, Hirzinger G. Motion tracking for minimally invasive robotic surgery. ITech Education and Publishing 2008; 117-48.

11. Chen D, Mayer E, Vale J, et al. A 3d stereo system to assist surgical treatment of prostate cancer. Workshop of Int Conf on Medical Image Computing and Computer Assisted Intervention. 2008; 9-17.

12. Noonan DP, Mountney P, Elson, et al. A stereoscopic fibroscope for camera motion and 3d depth recovery during minimally invasive surgery. IEEE Int Conf on Robotics and Automation. 2009; 3274-9.

13. Tsai R. An efficient and accurate camera calibration technique for 3d machine vision. Proceedings of Computer Vision and Pattern Recognition. Miami Beach, FL 1986; 364-74.

14. Bouguet JY. Bouguet calibration algorithm – unpublished, http://www.vision.caltech.edu/bouguetj/calib_doc

15. Tsai RY. A versatile camera calibration technique for high-accuracy 3d machine vision metrology using off-theshelf tv cameras and lenses. IEEE J Robotics and Automation 1987; 3: 323-44.

16. Woodham RJ. Photometric method for determining surface orientation from multiple images. Optical Engineering 1980; 19: 139-44.

17. Horn BK. Obtaining shape from shading information. Shape from shading, MIT Press, Cambridge MA 1989; 123-71.

18. Loh A, Hartley R. Shape from non-homogeneous, non-stationary, anisotropic, perspective texture. Proceedings of the British Machine Vision Conference 2005; 69-78.

19. Hartley RI, Zisserman, A. Multiple View Geometry in Computer Vision, second edition. Cambridge University Press 2004.

20. Salzmann M, Fua P. Deformable surface 3d reconstruction from monocular images. Synthesis Lectures on Computer Vision 2010; 2: 1-113.

21. Bradski GR, Kaehler A. Learning Open CV. 1st edition. O’Reilly Media, Inc. 2008.

22. Nguyen TN, Moseley JL, Dawson, et al. Adapting population liver motion models for individualized online image-guided therapy. Conf Proc IEEE Engineering in Medicine and Biology Society 2008; 3945-8.

23. Nguyen TN, Moseley JL, Dawson, et al. Adapting liver motion models using a navigator channel technique. Med Physics 2009; 36: 1061-73.