Computer Vision, OpenCV, UR5 Robot Arm, Manufacturing, Research, Intel Realsense

Overview

While getting my bachelor’s degree at Louisiana State University I worked as an undergraduate researcher in the Innovation in Control and Robotics Engineering Lab (iCORE Lab). My main area of research was computer vision for robotic systems. Specifically, as part of an NSF funded research project, I created a computer vision system to detect surface defects in fiber glass for autonomous sanding. Once, the locations of the defects were found, two types of path planning were used to create a sanding path. The path waypoints were then relayed to a UR5e 6 DOF robot arm with a sander attachment to sand the defected areas.

System Pipeline

System Components

Vision

Using traditional computer vision techniques like canny edge detection, morphological closing, contour detection, and binary mapping I was able to detect and isolate defects present on the surface of a fiber glass panel.

Computer Vision Pipeline

First, an Intel D435i camera captures the RGB and depth images of the fiber glass sample. A canny edge detector is applied to the RGB image to separate the inherit pattern visible underneath the surface of the fiber glass from the defects present on the surface. After this step, the surface defects become easily visible. To create a general region of the defective area, I use morphological closing to merge nearby edges and fill openings within a certain area to create a blob like structure where defects reside. Next I use contour detection to get the pixel locations of everything within the defective regions.

Path Planning

Once the locations of the defects are known, a path must be created so the robot can sand over all of the defective areas. Two different types of path planning were implemented. Multi-goal path planning was needed to make sure the robot goes to each of the separate defective regions and coverage path planning was used to ensure the robot sanded all of the defects within a given region.

For multi-goal path planning, a nearest neighbor algorithm was implemented to create a path that went to each region based off of the location of the regions centroid. Next, a grid-based sweeping algorithm is used to create a path that covers the entire area. Finally, the two plans are combined and the robot’s sander is pressed against the piece for the coverage paths to ensure sanding and it is offset from the piece during multi-goal movements so that smooth sections are not sanded erroneously.

Example of Finalized Path in 2D and 3D

Results

The system successfully detected the defected areas and we quantified the results by having a person manually label defects in one of the fiber glass image samples and compared that to what the system detected. Fifteen fiber glass panels were used and the calculated average sensitivity obtained was 66.24%, the average specificity was 78.20%, and the resulting accuracy was 81.02%. Furthermore, using a profilometer, I measured the surface roughness of certain defected areas before and after sanding and found that the average roughness at these areas was about half as rough after performing autonomous sanding.

Contributors

• William Ard
• Corina Barbalata
• Joshua Nguyen