In-vitro evaluation of stereoscopic liver surface reconstruction

Introduction

Tracking abdominal motion of organs is an important factor in image-guided navigation systems. The paper presents the evaluation methodology of a practical approach to measure liver motion, both respiratory and laparoscopic, with a tool guided in the operating room. The evaluation was based on the methodology described in the previous paper [1].

Aim

The proposed evaluation methodology can be divided into three stages: configuration of the stereo camera, camera calibration, solving of the correspondence problem. Our research aim was to survey the impact of the following three parameters on reconstruction accuracy: the number of calibration points, imprecision of monocular camera assembly, standard and high resolution images.

Material and methods

Reconstruction of point X in 3D space, called the world coordinate system, requires solving the correspondence problem from two monocular cameras [2]:



X = (PL, Pp, xL, xP),



where  denotes the triangulation algorithm, PL, PP are projection matrices of the left and right monocular camera, and xL, xP are coordinates of correspondence point in the left and right images.

The first step was to set up the equipment and conduct the calibration procedure. In our approach Tsai’s algorithm was selected as a calibration method [3, 4]. We tested Tsai’s method with different configurations of the calibration chessboards and different numbers of the calibration points. Both cameras should remain in the same relative position after the calibration procedure. The calibration procedure requires a few pose acquisitions of the calibration grid from both cameras, which is difficult during a laparoscopic procedure. It is impossible to use standard laparoscopic cameras to grab required poses directly in a patient position in the operating room. After taking out the cameras from trocars, cameras are reassembled in the same relative position to each other. Inaccuracy of this process influences projection matrices P, whose entities are decomposed as follows [2]:



P = K[R|t],



where: K – calibration matrix with internal camera parameters, R – rotation matrix in the world coordinate system, t – translation vector in the world coordinate system.

After calibration of a stereo camera and setting up the equipment (Figure 1) we focused on localization of correspondence points in the pair of monocular camera images.

The accurate reconstruction from 2 monocular videos require two types of adequacy:

• frame correspondence in time dimension,

• point correspondence in the image from both cameras.

The best solution is obtained by hardware synchronization. We rejected this approach because the assumption is to use standard laparoscopic cameras which do not make it possible. The proposed solution is based on time stamps of a video frame sequence, using time stamps from the AVI file format and the maximum time-shift threshold approach. The evaluation was done on exactly the same equipment that is used in the operating room: medical video processor Olympus Exera II CV-180 with constant value frame per second parameter, which defines our maximum time-shift threshold. Pairs of frames whose time-shift is bigger than threshold are rejected.

In the presented approach [1] two well-known methods for solving the correspondence problem were taken into account:

• RANdom SAmple Consensus (RANSAC) [3],

• Lucas-Kanade [5].

RANSAC and Lucas-Kanade require landmarks to be obtained in the pair of corresponding images. For evaluation of possible triangulation accuracy, we used artificial markers attached to an abdominal liver phantom (Photo 1). The defined marker pattern, as a characteristic configuration of white and black boundaries of the image areas, was applied to locate markers’ positions. Selecting correspondence points in pairs in both images was performed manually.

Results

The selected configuration of the calibration chessboard in our approach is presented below (Photo 2). Three calibration chessboard configurations were tested: both monocular cameras were calibrated on the left chessboard, both monocular cameras were calibrated on the right chessboard, the left camera was calibrated on the right chessboard and the right camera on the left one.

Generally we tested triangulation results on two data sets: left and right chessboard corners, liver phantom assigned landmarks.

The first data set allows an easy evaluation of results because we know the reference position of chessboards which define the world coordinate system (Photo 2). We use 30 mm chessboards with 81 corners. For the above-mentioned three calibration configurations the origin of the world coordinate system was fixed in the right corner of the left calibration chessboard (Photo 2). Moreover, we did the tested triangulation in SD/HD with 27, 45 or 81 points (which means: 3 upper rows of chessboard corners, 5 upper rows of chessboard corner rows and all chessboard corners respectively). The average residual mean square error (RMSE) in millimetres is presented below (Table I). There is anomaly in RMSE for 45 calibration points in HD resolution that is a little greater than for 27. However, this exception does not affect the tendency that with more calibration points we obtain higher accuracy (less RMSE).

The reconstruction positions of liver phantom markers were tested in three positions in the high resolution mode: reference position, translated position, rotated position.

The reconstructed position and the average RMSE in millimetres are presented below (Tables II, III). The reconstructed liver phantom landmarks in three positions are presented (Photo 3).

The comparison of results (Table I) indicates the advantage of the calibration in High Definition Mode over the calibration in Standard Definition. The suggested number of chessboard calibration points for Tsai’s method is more than 30 [2]. The performed tests highlight the decrease of RMSE for a higher number of calibration points, with one exception – discussed in the previous paragraph. More accurate results were obtained for calibration with one chessboard (left or right), where average RMSE is less that 0.5 mm, compared with calibrating cameras on different chessboards (in the worst case average RMSE is 1.61).

Applying two chessboards for calibration does not cause different results in RMSE for any of the chessboard corners.

Discussion

As regards the calibration chessboards setup by using one chessboard we assume that image noise is identically distributed in the observed coordinates, which generally is not true. In order to minimize triangulation RMSE in the whole reconstruction volume, it is advisable to select the set of calibration points from the whole volume. Since it is difficult to obtain, one calibration chessboard is used in practice [3]. There are some descriptions of research using a set of calibration grids instead of one grid [6]. We tested both approaches. The achieved results have less RMSE in the case of one chessboard. The impossibility to calibrate rigid configuration of two monocular laparoscopic cameras in the operating room (OR) patient position required reassembly of the camera setup, which caused RMSE increase in the case of two perpendicular chessboards. Taking OR circumstances into account, one calibration chessboard seems to be enough.

Tsai’s algorithm is a well-known solution for a camera calibration problem. It requires at least 11 calibration points [2], but normally between 20 and 60 are used. We applied two perpendicular, 30 mm calibration chessboards with 81 corners and tested 21/45/81 calibration points in both high and standard definition modes. The high camera resolution mode has less RMSE than the standard one. This is especially important if one calibration grid is used to calibrate both cameras (differences in RMSE between the high and standard mode are significant). The experiments proved the important impact of image resolution on triangulation accuracy.

The triangulation approach for surface reconstruction requires solving the correspondence problem. Two well-known approaches, RANSAC [5] and Lucas-Kanade [7], require localization of landmarks in pairs of corresponding images. For possible evaluation of triangulation accuracy we use artificial markers attached to the abdominal liver phantom. Our approach, which is based on a defined marker pattern, as a characteristic configuration of white and black boundaries of the image area, could not find all instances of markers (Photo 1). The reason for the omitted markers was a large difference from the defined pattern. To reconstruct a more complex liver surface we plan to use a Time of Flight camera for open surgery [8].

The sensitivity of the triangulation approach is a relevant issue in investigation on 3D reconstruction [9]. Measurements of differences in the reconstruction position by rotation of one out of two cameras at a few degrees in the operating room may cause a significant difference in reconstruction position of the liver surface (Photo 3). This is a challenge in the laparoscopic case where generally it is impossible to calibrate cameras in the same position as used for triangulation (patient position). The best solution for this approach is to use one stereo laparoscopic camera. There is one available solution which is a stereo camera in laparoscopic teleoperated robot-assisted surgery, da Vinci, but the price and long learning curve for an operating team are crucial obstacles for common use. Nowadays there is a lack of cheaper and implemented medical solutions from competitive companies.

Acknowledgments

The authors would like to acknowledge the financial support from the Polish National Centre for Research and Development (NCBiR), grant no. LIDER/ 03/47/L-1/NCBiR/2010: “Development of planning system and computer aided minimally invasive surgery for hepatic and metastatic liver carcinoma localization, diagnosis and destruction”.

References

 1. Spinczyk D, Karwan A, Rudnicki J, et al. Stereoscopic liver surface reconstruction. Videosurgery Miniinv 2012; 7: 181-7.

 2. Hartley RI, Zisserman A. Multiple view geometry in computer vision. Cambridge University Press. New York, USA 2008.

 3. Tsai RY. A versatile camera calibration technique for high-accuracy 3d machine vision metrology using off-theshelf tv cameras and lenses. IEEE J Robotics and Automation 1987; 3: 323-44.

 4. Bradski GR, Kaehler A. Learning OpenCV: computer vision with the OpenCV Library. O’Reilly Media, Inc. Sebastopol, USA 2008.

 5. Fischler M, Bolles R. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communication of the ACM 1981; 24: 381-95.

 6. Heikkilä J. Geometric camera calibration using circular control points. IEEE Transactions on Pattern Analysis and Machine Intelligence 2000; 22: 1066-77.

 7. Lucas BD, Kanade T. An iterative image registration technique with an application to stereo vision. Conf Proc of the 7th Int Joint Conf on Artificial Intelligence 1981; 674-9.

 8. Schaller C, Penne J, Hornegger J. Time-of-Flight sensor for respiratory motion gating. Med Phys 2008; 35: 3090-3.

 9. Bier A, Luchowski L. Error analysis of stereo calibration and reconstruction. Proc of the 4th Int Conf on Computer Vision/ Computer Graphics Collaboration Techniques MIRAGE 2009; 230-42.



Received: 14.05.2012, revised: 21.10.2012, accepted: 12.11.2012.