Stereo_Camera Calibration error

[bensailor]

Hi,

I am running the Stereo Pi Fisheye tutorial. I am on part 4, the calibration. I seem to calibrate both individual cameras but it fails on the Stereo calibration.

File "4_calibration_fisheye.py", line 170, in calibrate_one_camera
result = calibrate_one_camera(objpointsRIght, imgpointsRight, 'right')
File "4_calibration_fisheye.py", line 170, in calibrate_one_camera
(cv2.TERM_CRITERIA_EPS+cv2.TERM_CRITERIA_MAX_ITER, 30 1e-6)
cv2.error: OpenCV(3.4.3) /home/pi/opencv-python/opencv/modules/calib3d/src/fisheye.cpperror:(-215:Assertion failed) fabs(norm_u1) > 0 in function 'InitExtrinsics'

Any ideas what could be going wrong? Is it the pictures?

Could it be there is a delay between cameras? How to check for this?

[bensailor]

i have taken the photos again and re run the script and the error has changed
Left camera calibration...
Right camera calibration...
Stereoscopic calibration...
./calibration_data/240p/camera_calibration_left.npz
['camera_matrix', 'distortion_coeff', 'imgpoints', 'map1', 'map2', 'objpoints']
Camera calibration data has been found in cache.
./calibration_data/240p/camera_calibration_right.npz
['camera_matrix', 'distortion_coeff', 'imgpoints', 'map1', 'map2', 'objpoints']
Camera calibration data has been found in cache.
Calibrating cameras together...
Traceback (most recent call last):
File "4_calibration_fisheye.py", line 299, in <module>
result = calibrate_stereo_cameras()
File "4_calibration_fisheye.py", line 257, in calibrate_stereo_cameras
cv2.CALIB_FIX_INTRINSIC, TERMINATION_CRITERIA)
cv2.error: OpenCV(3.4.3) /home/pi/opencv-python/opencv/modules/calib3d/src/fisheye.cpp error: (-215:Assertion failed) abs_max < threshold in function 'stereoCalibrate'

[Realizator]

Hi Bensailor,
I mentioned this problem in my last article.
The reason is one of your images. You should find this image and exclude it, and I use "halving" method for this. If you will have a problem with this, please send me your images set, and I will try to localize it.
Looks like it is often problem, and I need to try to catch this error and find wrong image automatically (but this is not easy).
UPD> Your left and right cameras are definitely swapped on your image...

[bensailor]

Hi

I fixed the swapped images, and tried the halving technique, still no success as when i decrease number of photos it claims bad matrix's.

Do you have a place where i can send photo sets?

thanks

Ben

[Realizator]

You can upload it to the cloud (like Google Drive) and share a link.

[bensailor]

Link to my photos below. Thank you very much. I look forward to hearing back from you I have had no success locating the bad photo.

https://drive.google.com/open?id=1Rgh9I ... 8iQierr70s

[Realizator]

I've downloaded your photos.
I don't know, if you try to turn on checkerboard visualization here. I always turn it on for the first run.
So as I can see chessboard was not found at all. I.e. there are no any images in your set, where OpenCV were able to find checkerboard.

Here are some tips:
- Light. You have a window on the left, and have a "counter light". Light should go from your camera and highlights chessboard, but not vice versa. Try to orient your camera to the area without bright lights
- Size of the chessboard on your images. You use 200 degree camera, and checkerboard looks tiny. Try to keep it closer to the camera. You can look at a sample image set from our repository to estimate acceptable size of the chessboard on the image.
- I noticed your chessboard image is not flat. This will not stop corners detection algorithm, but may affect on the calibration quality.
(For more tips please read part "4. Let's calibrate.." in our article.)

As a first step I'd recommend you to make another session with no counter light, and with keeping checkerboard closer to the camera.
Next step (if calibration is still not good) - flatten checkerboard setup.

You can also use advanced approach. As I mention in our article, we use a couple of 320x240 images for the depth map, but 640x480 for calibration. You can use a couple of 1280x960 images for calibration - in this case corner finding process will be much more successful. You need to look at this piece of code for this trick.

[bensailor]

Thank you for the tips. I will take these on board and try again.

I find it strange though when I ran the code it claims to detect checkerboard and successfully calibrate both left and right camera just not stereo.

Thanks again I will try this again.

[Realizator]

Hi Bensailor,
Are you sure you're uploaded right the same images you used for calibration? I've tested your set on both OpenCV image and on MacOS with OpenCV and Python, and checkerboards are not detected. I'm curious to understand, where our approaches are different and why your code is able to detect them.

[bensailor]

Hi,

Sorry for long reply been away, and its been chaotic with current situation. I now have some time and really want to solve this.

One thought i had with why you cant detect checkerboards is i used a 8x6 checkerboard. Sorry i should have mentioned this before.

I have tried again, with a bigger 8x6 checkerboard and properly stuck to hard cardboard and better lighting, but still run into same problem with able to calibrate left and right but not stereo pair.

Also i have the same code as you but it doesnt display any images, which i find strange

Cheers,

Ben

[bensailor]

I have solved the displaying image problem

[bensailor]

I have fixed my stereo calibration camera issue.
It seems i had a few bad stereo pairs, using the example mentioned here and the fix I was able to solve it:
https://answers.opencv.org/question/213 ... calibrate/

Adding this code to calibration after corner detection and before scaling I was able to detect bad stereo pairs and flip the corners.
if (retL == True) and (retR == True ):
diff = cornersL - cornersR
lengths = np.linalg.norm(diff[:, :, 1], axis=-1)
sum = np.sum(lengths, axis=0)
if (sum > 2000.0):
print("THIS STEREO PAIR IS BROKEN!!! Diff is: "+str(sum))
cornersR = np.flipud(cornersR)

[Realizator]

Wow, thank you for sharing the solution!
I planned to use another approach (catching exception), but your solution is more elegant. I will test it with my code and update GitHub repository!

UPD> Just a curious, where "2000" comes from, and how this number should be changed according to the calibration images resolutions...

[vram]

Hi,

I am new to stereopi.

I just started with the steps suggested for package "stereopi-fisheye-robot-master". I have come across the same error reported in the beginning of this thread while trying to calibrate for the same set of images given under "scenes" folder in the original package.

As I have not introduced any new image, I am confused if I am missing out anything.

The error is:

Traceback (most recent call last):
File "/home/pi/stereopi-fisheye-robot-master/4_calibration_fisheye.py", line 299, in <module>
result = calibrate_stereo_cameras()
File "/home/pi/stereopi-fisheye-robot-master/4_calibration_fisheye.py", line 257, in calibrate_stereo_cameras
cv2.CALIB_FIX_INTRINSIC, TERMINATION_CRITERIA)
cv2.error: /build/opencv-L65chJ/opencv-3.2.0+dfsg/modules/calib3d/src/fisheye.cpp error: (-215) svd.w.at<double>(0) / svd.w.at<double>((int)svd.w.total() - 1) < thresh_cond in function CalibrateExtrinsics

Can you please suggest how to overcome this?

Thanks,
Ram

[stereomaton]

I do not have my stereopi here, but since you said that you used only the data of the repo, I tested on my PC.
I had a guess on what could happen and did this test:

1) Execute 4_calibration_fisheye.py directly after a fresh clone. I get the same error.
2) Execute the way I would have, that is execute first 3_pairs_cut.py (1 & 2 are for test camera and capture) then 4_calibration_fisheye.py. It works.

[vram]

Thanks Stereomaton for the response.

As suggested, I tried running step 3 first followed by step 4 on a fresh repo. I continue to get the same error.

In fact, I first observed this issue with my own captured scenes (following steps 2 to 4). As there was a suggestion that there could be issues with some of the captured images, I first wanted to confirm that the SW works fine with the reference pictures but ended up having the same issue.

I have tried it out with a fresh repo separately on raspberry as well as Windows system using python 3.7. Issue exists in both.

[Realizator]

Hello Vram,
Sorry for my late reply. I will try to check the reference code and images to find this issue and fix the code to avoid it in a few days!

[Realizator]

Vram, I repeated all experiments. Looks like I didn't upload the latest fixes.
For the current set, we have a few photos, where chessboard corners are too close to the side. This is an undocumented requirement for the calibration session. I'm uploading the file with the draft fix, please check if it works for you. If everything is ok, I will polish this code and upload it to Github.

I've added a row 101 (getting a size of the image loaded), and check for the distance of the closest corners to the 0 of X and Y axis (rows 123-150).
You can adjust the sensitivity of this filter in rows 135-136:

Code: Select all

          border_threshold_x = loadedX/7
          border_threshold_y = loadedY/7

For example, for our photo sequence it means our loadedX = 640, and the border_threshold = 91.4. So, if the distance from the cell corner to the 0 of the X axis is less than 91.4, this pair is ignored.
I commented the checking of the distance to the end of the axis (you may found comments in the code), as looks like this not causing our problem.

[vram]

Realizator, with the fix provided, I am able to complete calibration for the reference images. I don't see the previously reported error any more. Thank you.

Let me try it out with my own sequence of images and calibrate. Once that goes through fine, let me confirm you.

BTW, I am trying to use this package (fisheye) along with V2 camera for depth measurement. Hope it is not a must to use wide angle cameras here. Please confirm.

[Realizator]

Hello Vram,
This fisheye code can work with both fisheye and non-fisheye images. In this forum thread, I used this fisheye code with the customer's set, taken on V2 non-fisheye camera. You can see it works, and even does a minor barrel undistortion for V2 camera.