Internship Report on Peg-In-Hole

Assembly

By- Aman Jain

Submitted to:
Dr. Riby Abraham Boby
Department of Mechanical Engineering
IIT JODHPUR
Abstract
The peg-in-hole assembly is a fundamental task in robotic automation, representing a
benchmark for evaluating precision and control in robotic systems. This project focuses
on implementing and optimizing a peg-in-hole assembly using PyBullet, an open-source
physics engine. The goal is to achieve accurate and efficient alignment and insertion of a
peg into a hole by utilizing advanced control algorithms and simulation techniques. The
simulation provides insights into the dynamics of the task and helps identify the factors
affecting performance.

Acknowledgment
I would like to express my sincere gratitude to my esteemed mentors during the IIT,
Jodhpur Summer Internship. I extend my heartfelt appreciation to Dr Riby sir and
Shameer sir for their unwavering guidance, insightful expertise, and invaluable support
throughout my learning journey. Their dedication and knowledge have been
instrumental in shaping my understanding of the industry matter and enhancing my
skills. I am truly thankful for their time, effort, and commitment in making this
internship training a transformative experience.

1
Contents

1 Introduction 3

2 Literature Survey 4
2.1 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.1.1 Object Picking-up . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.1.2 Assembly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Analytical Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3 Simulation of Peg-In-Hole Assembly 8

4 Results and Observations 11

5 Conclusion 13

2
Chapter 1

Introduction

Robotic assembly tasks, such as the peg-in-hole insertion, are critical in manufacturing
and automation industries. These tasks require high precision and control, which can be
challenging to achieve due to uncertainties and variations in the environment. The peg-
in-hole problem has been widely studied as it encapsulates many of the challenges faced
in robotic manipulation, including precise positioning, force control, and error correction.
This project aims to simulate the peg-in-hole assembly using PyBullet and to explore the
methods for optimizing the process.

3
Chapter 2

Literature Survey

2.1 Methodology
2.1.1 Object Picking-up
In cluttered environments, conventional object detection methods relying on shape, color,
or edges may struggle due to fluctuations in environmental characteristics like lighting
and object density. In such scenarios, grasping tasks require not just detection but also
object recognition. Machine learning techniques, particularly deep learning, have shown
superiority in dynamic and cluttered environments, offering reliable performance for robot
grasping by leveraging advanced learning algorithms.

A. Training the Model

They took lots of pictures of objects from different angles, even when some objects were
partially hidden. Then, they adjusted the pictures to a standard size and made them
look a bit different to help the computer learn better. The computer was trained to
recognize objects in these pictures, with special software keeping an eye on how well it
was learning.

B. Model Implementation
Once the computer was trained, they put it to work. They used a RGB camera to spot
black cylinders among other objects. When the computer found one, it told the robot
where the cylinder was.Then, using some clever coordinate math, they figured out where
the robot should go to pick up the cylinder.

4
 Figure 2.1: Object Picking Process

2.1.2 Assembly
After the cylinder is picked up, the robot moves to the assembly area. The position of
the hole is then determined using sensor information. The robot will accomplish the final
localization and assembly using visual and force servoing. Due to the inaccuracies in the
camera calibration, robot kinematics, etc., this is done in three main stages –

A. Rough Localization
Rough localization wherein the hole center position is detected when the robot is over
the assembly region.

B. Fine Localization
Fine localization using sensor information after positioning the robot closer to the plane
of assembly according to the rough estimate obtained earlier.

C. Continuous Localization
Continuous localization that involves contact with the assembly plane and obtaining
continuous visual feedback while in force control mode.

5
 Figure 2.2: Rough and Fine Localization

Figure 2.3: Assembly Process

2.2 Analytical Analysis

Figure 2.4: Peg in contact with hole partially

6
 Ot : Represents the center of the peg.
R : Represents the point at which the reaction force Fr acts.
Fr : The reaction force acting at point R due to contact with the hole.
Fa : Force applied by the robot on the planar surface on which holes are located.
T : The torque generated due to the reaction force.
Pr : The position vector from the center of the peg O to the reaction force point R.
Os : The coordinate system for the peg-in-hole task.

Algorithm in Process-

1. There is an equilibrium due to the presence of an action and reaction force. The
force Fa is applied by the robot on the planar surface on which the holes are located.
Whereas, Fr is the reaction force. The action force can be assumed to be applied
along the Z axis. The corresponding reaction force can be assumed to act at point
R. The moment about Ot due to Fr can be expressed as -

m = pR × f R (2.1)

2. Note that Z axis of the F/T measurement frame is aligned with the axis of the peg
along which force is applied. It can then be considered that the moment about
Z axis of the sensor vanishes i.e Mz =0. Also, Fx =0 and Fy =0, which gives the
following expression:    
mx 0 fRz −fRy
= pR (2.2)
my −fRz 0 fRx

3. Since Mx ,My , Fr can be easily measured by the FT Sensor , we can easily calculatePr .

4. Lateral Misalignment: If Pr has components in the x or y direction, it indicates

lateral misalignment. The robot can correct this by moving the peg in the opposite
direction of Pr ’s x or y components.

5. Rotational Misalignment: The measured moment m can also indicate rotational

misalignment. If there is a significant moment, it suggests the peg needs to be
rotated to reduce the moment to zero.

6. Feedback Loop: The robot uses the real-time data from the sensor to continuously
adjust the peg’s position. By minimizing the components of Pr and the moment
m, the peg is brought into alignment with the hole.

7. Iterative Adjustments: The robot makes iterative adjustments to the peg’s position
and orientation based on the updated measurements of Fr and m. The goal is to
reduce the forces and moments, indicating proper alignment.

7
Chapter 3

Simulation of Peg-In-Hole Assembly

Algorithm Proposed for Simulation-

A. Initialization:

1. Connect to PyBullet and set up the simulation environment.

2. Load the plane, peg, and hole URDFs.

3. Define initial positions, orientations, and dynamics properties for the peg and hole.

B. Simulation Loop:

1. The simulation runs for a maximum number of steps or until the peg is successfully
inserted.

2. Getting Positions and Orientations: In each simulation step, retrieve the current
position and orientation of the peg and the hole.

3. Contact Information: Check if there are any contact points between the peg and
the hole. If contact points exist, apply tilt correction.

4. Tilt correction is crucial for ensuring that the peg aligns correctly with the hole
before insertion. The key concepts here are the lever arm and torque.

• Lever Arm: The distance vector from the center of the peg to the contact point.

• Contact Force: The force exerted at the contact point due to interaction between
the peg and hole.

• Torque: The rotational force calculated as the cross product of the lever arm and
the contact force.

C. Disconnect from Pybullet server.

UDRF Structure (Explanation)

URDF (Unified Robot Descripton Format) is an XML format for representng a robot
model. It is widely used in robotics simulaton frameworks like PyBullet, Gazebo, and
ROS (Robot Operating System). A URDF file describes the structure of a robot, in-
cluding its links (bodies), joints, visual representation, collision geometry, and physical

8
properties such as mass and inertia. In simulaton environments like PyBullet, URDF
files are used to define the physical and visual properties of robots or objects. This allows
for accurate physics simulatons and visualizations. URDF is a standardized format for
describing robots, which makes it easier to share and reuse robot models across different
simulation and robotics platforms. PyBullet can load these URDF files to create the
corresponding simulated objects.

1. <robot>: Defines the root element of the URDF file, representing a robot model
named=”xyz”.

2. <link>: Defines a rigid body.

3. <visual>: Visual representation with position, geometry, and material.

4. <collision>: Collision properties with position and geometry.

5. <inertial>: Inertial properties with position, mass, and inertia tensor.

Python Script (Explanation)

1. The script imports the necessary libraries:

• pybullet: Used for simulation.

• numpy: For mathematical and vector calculations.
• time: For time control in the simulation loop.
• pybullet data: To load URDF models.

2. The code connects to the graphical user interface (GUI) of PyBullet, sets the path
to search for URDF files, and initializes the gravity for the simulation environment.

3. The simulation environment is established:

• The ground surface (plane) is loaded.

• The starting position for the peg and hole is specified.
• The peg and hole URDF files are loaded into the environment at the defined
positions.
• Friction and restitution (bounciness) properties for the peg and hole are set to
define their physical interactions.

4. Camera parameters are configured to set the view for the simulation.

5. The maximum number of simulation steps is established, along with the duration of
each simulation step, corresponding to a simulation frequency of 240 Hz. Precision
required for aligning the peg with the hole in the XY plane is defined, and gain
values determine how much the peg’s position and orientation should be adjusted
during each step.

6. Contact info contains details about contact points between the peg and hole,
including:

9
 • Contact position: Retrieves the contact position (the point where the peg
and hole touch) and converts it into a NumPy array for easier mathematical
operations.
• Contact normal: Calculates the contact normal (the direction of the force at
the contact point) and converts it into a NumPy array.
• Contact force: Extracts the magnitude of the contact force.

7. The lever arm is computed as the vector from the peg’s center of mass to the
contact point. Torque is calculated based on the contact force using the cross
product of the lever arm and the force vector. This torque is used to correct the
peg’s orientation.

8. The peg’s orientation is converted from quaternion to Euler angles using Peg orn euler.
This makes it easier to apply corrective rotation. The Corrective rotation is
computed to counteract the torque. The orientation gain is a tuning parameter
that determines the strength of the corrective rotation. The orientation is adjusted
as follows:
peg orn euler+ = corrective rotation

9. The adjusted orientation is then converted back from Euler angles to a quaternion.
PyBullet uses quaternions internally to avoid problems such as gimbal lock. The
new peg orientation, along with the contact force and torque, is returned for further
use in the simulation.

10. The simulation loop focuses on alignment in the XY plane and insertion. It includes
additional force and torque calculations, as well as the use of a tilt function to adjust
the peg and apply an external force for final insertion.

11. Finally, the script disconnects from the PyBullet server, completing the simulation
process.

More details and code can be found in the repository : GitHub Link.

10
Chapter 4

Results and Observations

Observations
1. It is clearly observed on the terminal output that initially when no contact is made
the contact force and torque is zero but as soon as the contact is made we are
getting some value of contact force and torque on the terminal output.

Figure 4.1: Terminal Output

2. The applied force and torque on the peg provide insights into how the peg aligns
itself with the hole.The use of contact forces to adjust the orientation of the peg
demonstrates the effectiveness of tilt correction produced by reaction force in real-
time.
3. The step-by-step alignment process, both in the XY-plane and along the Z-axis,
shows the gradual approach of the peg towards the hole. This demonstrates the
importance of controlled movements to avoid collisions and ensure proper insertion.
4. Observing the contact points and the forces at these points highlights the critical
areas where adjustments are necessary. It was noted that as the peg approaches the
hole, the contact forces play a significant role in guiding the peg into the correct
position.
5. The defined thresholds for alignment (XY and Z) are crucial for determining when
the peg is considered successfully inserted. These thresholds help in assessing the
accuracy and efficiency of the insertion process.

Results
1. Successful Alignment with hole: The simulation successfully achieves the alignment
of the peg with the hole. This is validated by the position and orientation of the
peg matching the alignment thresholds.

11
 Figure 4.2: Approach Figure 4.3: Contact Figure 4.4: Tilt cor-
towards hole with hole and tilt oc- rection and final align-
curs ment with hole

2. The peg’s orientation adjustments based on contact forces demonstrate the sys-
tem’s ability to dynamically respond to environmental interactions. This results in
smooth and precise insertion without manual intervention.

3. The real-time feedback on peg position, orientation, contact force, and torque pro-
vides valuable data for analyzing the insertion process. This data helps in fine-
tuning the simulation parameters for better performance.

4. The simulation is efficient in terms of computational steps, with the peg being
aligned successfully within a few thousand steps. This indicates that the approach
and parameters used are effective for such tasks.

12
Chapter 5

Conclusion

The peg-in-hole simulation using PyBullet has provided a comprehensive under-

standing of the dynamics involved in such an assembly task. Overall, this simulation
has demonstrated the effectiveness of using PyBullet for modeling and analyzing
peg-in-hole assembly tasks. The observations and results obtained provide a solid
foundation for further research and development in automated assembly systems.

13
Bibliography

[1] “Robotic Pick and Assembly Using Deep Learning and Hybrid Vision/Force
Control(2021)”. Link

[2] “An Approach for Peg-in-Hole Assembling using Intuitive Search Algo-
rithm based on Human Behavior and Carried by Sensors Guided Industrial
Robot(2015)”. Link

[3] “Peg-in-Hole Assembly.” ScienceDirect (2021). Link

[4] “Implementation of YOLOv3 Simplified.” Analytics Vidhya (2021). Link

[5] “The Proposed Peg-in-Hole Assembly: A Search Phase and an Insertion Phase.”
ResearchGate (2019). Link

14