Research

Our research goal is to develop robust machine vision algorithms for robotic automation and intelligence under challenging unstructured environments. To this end, we explore and conduct research on the topics of Visual Servoing, Autonomous Driving, Soft Robot, Unmanned Aerial Vehicles(UAVs), Medical Robot, Reinforcement-learning Control, Multi-robot Control and Large-scale Scheduling and Machine Vision Projects.

Visual Servoing

We concentrate on visual servoing control in unstructured environments. The depth-independent interaction matrix is proposed to decouple nonlinearity from depth, based on which the adaptive law is proposed to estimate unknown camera parameters online. The controller can be consequently designed using an uncalibrated monocular camera, realizing regulation and tracking performance. The proposed visual servoing techniques have been employed in the control of diverse types of robot platforms, including industrial robots, soft robots, flexible robots, mobile robots, etc.

hpp Visual Servoing of Robot Manipulator
In vision-based robotic manipulation, it is usually required that the object should present a desired shape or be viewed at a specific angle to facilitate subsequent operations such as object crawling and component inspection. To implement this kind of visual servoing tasks, a novel image feature based on curve parameters of Bezier and nonuniform rational B-spline (NURBS) curves are designed. An adaptive depth-independent controller is designed to estimate the unknown curve parameters as well as the depth information online (TMECH 2018). In addition, based on the theory of visual servoing, we have developed an applications, interaction with bottle-like objects, which can be used to deburring, polishing, and welding the inner surface of the bottle-like mold. Based on the geometry of the object, a new generalized constraint called the bottleneck (BN) constraint is proposed, which ensures the tool passes through a fixed 3-D region and avoid collisions with the boundary of the region. A novel dynamic controller is designed to realize the hybrid vision/force control under the BN constraint (ICRA 2021).
hpp Visual Servoing of Flexible Manipulator
We propose a novel image-based visual servoing for a flexible aerial refueling boom with an eye-in-hand camera. The dynamic model of the flexible refueling boom is decomposed into a slow subsystem and a fast subsystem based on the singular perturbation approach. With respect to slow subsystem, the image feedback is used to control the flexible refueling boom so that the projection of the point marker on the back of the receiver converges to the desired position. With respect to fast subsystem, linear quadratic regulator (LQR) is applied to stabilize the vibration of the flexible refueling boom. The asymptotic convergence of the image error to zero is verified based on the Lyapunov theory. Simulation is used to demonstrate the effectiveness of the proposed method. (TMSC 2020)
hpp Visual Servoing of Wheeled Robot—Regulation
We propose novel image-based visual servoing schemes for the pose stabilization and position control problem of mobile robots with an overhead fixed camera. A new image-based kinematic model is introduced, removing camera intrinsic and extrinsic parameters from the image Jacobian matrix and making the design of camera-parameter independent image-based controllers possible. In the proposed schemes, neither accurate nor approximate knowledge about the camera intrinsic and extrinsic parameters is required, and the totally unknown camera can be mounted on the ceiling with an arbitrary pose, which can make the controller implementation very simple and flexible. (TRO 2019, TAC 2018)
hpp Visual Servoing of Wheeled Robot—Trajectory Tracking
In real applications, it is very important to control the mobile robot to move along a desired trajectory to a desired position, in order to achieve obstacle avoidance or keep the mobile robot in the camera field of view during the control process, either of which is known to be the key to success of task execution. To this end, we propose a new calibration-free image-based trajectory tracking control scheme for nonholonomic mobile robots with a truly uncalibrated fixed camera. By developing a novel camera-parameter-independent kinematic model, both offline and online camera calibration can be avoided in the proposed scheme, and any knowledge of the camera is not needed in the controller design. The proposed trajectory tracking control scheme can guarantee exponential convergence of the image position and velocity tracking errors. (TASE 2020)
hpp Visual Servoing of Wheeled Robot—Formation Control
In many existing formation control approaches for nonholonomic mobile robots, the leader velocity is required to be measured and transmitted to the follower for controller design. To make it applicable to the environments where providing the robots with the capability of global positioning is difficult or impossible, we should develop formation controllers without measurement of the robot global position information. To this end, we develop novel continuous formation controllers for mobile robots without measurement of the leader velocity such that communication between the mobile robots is not required. To address the unavailability issue of the leader velocity, observers based on adaptive control technique are proposed to obtain estimation of the leader velocity from information of the follower’s onboard sensors. The effect of the velocity estimation error on the closed-loop stability is considered in the stability analysis based on Lyapunov stability theory, and it is shown that global stability of the combined observer–controller closed-loop system is ensured by the developed approaches. (TMECH 2020, TRO 2018)

Autonomous Driving

For autonomous driving, our research mainly focusses on the perception and localization through learning-based algorithms. Specifically, we aim to develop robust AI-based Simultaneous Localization and Mapping (SLAM) system for autonomous driving and mobile robotics under challenging environments based on multi-sensor perception. Our work includes learning-based odometry, large-scale mapping, long-term loop closure and relocalization.

hpp hpp Outdoor Visual and LiDAR-based Odometry
Due to noisy pixels caused by dynamic objects in the outdoor scenes, visual odometry is heavily affected under such dynamic environments. We propose a confidence-based unsupervised visual odometry model to fully leverage the similarity and consistency of correspondent pixels, improving the robustness to dynamic objects (TITS 2021, IROS 2019). Since motion segmentation is also important to dynamic scenes, we extract features from sequential depth maps through GRU to segment dynamic objects from visual perception, which improves the odometry performance on KITTI dataset and real-site application (PRICAI 2019). For LiDAR-based odometry, we introduce a novel 3D point cloud learning model, named PWCLO-Net, using hierarchical embedding mask optimization. It outperforms all recent learning-based methods and outperforms the geometry-based approach, LOAM with mapping optimization, on most sequences of the KITTI odometry dataset (CVPR 2021).
hpp Large-scale Depth Estimation and Mapping
In autonomous driving, the perceptions of depth, optical flow and camera ego-motion are the fundamental abilities for many high-level tasks, such as SLAM, obstacle avoidance and navigation. We studied the problem of pixel mismatch caused by occlusion and illumination change, the problem of depth degradation in long-term training, and the problem of optical flow estimation in occlusion area. By analyzing tasks above, pixels in the middle frame are modeled into three parts: the rigid region, the non-rigid region, and the occluded region. After explicitly occlusion handling, several strategies are proposed to handle the problems in different regions, such as the less-than-mean mask, the maximum normalization, and the consistency of depth-pose and optical flow. Through the proposed strategies, the performances on the three tasks are improved, which is demonstrated in public autonomous driving datasets. (TITS 2020, ICRA 2019)
hpp hpp Long-term Loop Closure and Relocalization
The changing environments pose great challenge on the long-term SLAM algorithms, especially for loop closure and relocalization. We A self-supervised representation learning is proposed to extract domain-invariant features through multi-domain image translation by introducing feature consistency loss. Besides, a novel gradient-weighted similarity activation mapping loss is incorporated for high-precision localization (JAS 2021, IROS 2019). To leverage the high-quality virtual ground truths without any human effort, we propose a novel multi-task architecture to fuse the geometric and semantic information into the latent embedding representation through syn-to-real domain adaptation(TIP 2020). For the large-scale point cloud from LiDAR, we propose a novel discriminative and generalizable global descriptor to represent the large-scale outdoor scene, which reveal the continuous latent embedding feature space for place recognition and loop closure. Based on LPD-Net, point cloud registration is implemented for 6-DoF pose regression and relocalization after loop closure (ICCV 2019, IROS 2019, IROS 2020).
hpp Unmanned Delivery Robot
Cooperating with Vipshop, we proposed a multi-sensor fusion based unmanned system with autonomous navigation, localization, planning and control algorithm to solve the last-mile delivery problem in the logistics parks. Our unmanned system integrates multi-sensor fusion based SLAM, multi-model perception, dynamic path planning, obstacle avoidance algorithm and motion control algorithms, achieving high-precision mapping, localization, perception, navigation and obstacle avoidance under complex environments. The system has been validated on multiple platforms and accurate vision-based navigation, localization and fixed parking tasks have been accomplished in the large-scale challenging outdoor environments. And it has been successfully applied to the industry field of logistics and distribution and the trial operation in the SJTU campus and Vipshop headquarter has been completed as well. The single delivery path is more than one kilometer long with satisfying operation effects, and the cumulative delivery has been about thousands of express items.

Soft Robot

For soft robots, our research mainly focuses on design, modeling and control for soft robots that are made up of silicone rubber and electroactive polymers. To be specific, we have been working on a soft manipulator that is dedicated to surgery in a minimally invasive fashion and a soft gripper that is able to grasp fragile objects. We have also developed new designs for soft robots capable of multi-modal locomotion for the exploration of unknown environments.

hpp hpp Design of Soft Manipulator and Soft Gripper
A cable-driven soft manipulator system is designed for cardiac ablation in a minimally invasive manner. The system is totally made of soft materials and has no rigid structures inside. A shape sensor network based on Fiber Bragg Gratings (FBGs) is embedded inside the soft manipulator to obtain the robotic shape in real-time. A proximity sensor, which consists of a 4-point sensor array, is affixed at the end of the soft manipulator for tracking a beating heart. Shape memory polymer (SMP) has been successfully integrated with soft robotics for resolving compliance complexities associated with poor stiffness or low payload capability. However, heating of thermally responsive SMP has always been challenging for robotics applications. To overcome this challenge, we fabricated a soft robotic finger and introduced the concept of artificial joints by combining SMP substrate with attached heaters to a soft pneumatic finger and achieved different bending motions by activating different sets of joints. For soft gripper sensing, we present a novel design and fabrication procedure of a soft robotic actuator that has the attributes of pressure and curvature sensing. (ROBIO 2018, CYBER 2018)
hpp Kinematics and Dynamics for Soft Robots
Due to the distinction of the modelling between rigid robots and soft robots, a new framework that concerns kinematics, statics and dynamics for soft robots is badly needed. To do this, we propose three-dimensional dynamics by combining geometrically exact Cosserat rod theory and the Kelvin model. Both curvature and strain are taken into accounts on the basis of the piecewise constant curvature model. Based on this, an underwater dynamics considering friction model is proposed that adopts the Column friction model to compensate for the actuation force’s loss in the transmission process. The dynamics presented can adapt to variable environments and serve as the platform for the controller design (TMECH 2018). Based on the solved system model, we have developed model-based force and collision detection algorithm (ICRA 2021).
hpp Soft Robot Control in the Free Space
Owing to the modelling differences between rigid robots and soft robots, control algorithms for rigid robots cannot be directly applied to soft robots. Therefore, we have been dedicating ourselves to controller design specific to soft robots. We proposed series of visual servo controllers, such as an adaptive visual servoing controller considering special optical conditions and environmental disturbances (TIE 2019, TMECH 2019), shape control leveraging shape features to solve the feature correspondence problem of the continuum robot (RA-L/ICRA 2021); and fault-tolerant control merely relying on a monocular camera by meticulously designing signals into the dynamic process to trigger divergent performance (TIE 2020).
hpp Soft Robot Control in the Constrained Environments
To improve the controllability of the soft robot in the constrained environments, we have conducted an array of research into interaction with environment. We proposed a hybrid vision force controller based on a deformation model, leveraging the real-time deflection model updating techniques and realizing accurate force interaction performance (TCST 2019). To address the control problem with high safety level, e.g., applications in robot-assisted surgery, we should sometimes satisfy the dual requirements of accurate positioning of the end effector and non-collision with the organs in the body. To this end, we proposed a hybrid controller aiming at simultaneous obstacle avoiding and visual servoing of the robot extremity (TMECH 2020)
hpp Design of Small-scale Multi-modal Locomotion Soft Robot
Despite the adaptability potential of small-scale soft robots in unknown terrains, their performance often gets hindered due to inherent restrictions in soft actuators and compact bodies. To tackle this issue, we have designed a rapid-moving soft robot that is powered by electroactive materials. This robot merges the benefits of dielectric elastomer actuators (DEAs) and shape memory alloy (SMA) spring actuators, which paves the way for its high-performance multi-modal locomotion. We have developed nonlinear models for both DEAs and SMAs to analyze the robot's performance. The design parameters of the robot were fine-tuned based on these models to enhance its running and jumping capabilities. The performance of the designed robot was experimentally tested, which showed its excellent performance in running speed and jumping height. Meanwhile, the robot's turning motion and jumping angle can be controlled by the coordinated actuation of different actuators.

Unmanned Aerial Vehicles(UAVs)

For unmanned aerial vehicles, our research mainly focuses on image-based control and trajectory planning. Because GPS can be unqualified in indoor or in cluttered urban areas, also unreliable at low altitudes, we aim to control the UAV to perform servoing or tracking tasks by taking advantage of the visual information only providing by a monocular camera. On the other hand, we are also interested in generating safe and dynamically feasible trajectories for UAVs in the obstacle-cluttered environment. Our work consists of image-based visual servoing, image-based visual tracking, and real-time trajectory generation.

hpp Visual Servoing of UAVs
Due to the underactuation and the nonlinearity of the quadrotor dynamics, we use properly defined image features to design an IBVS controller for a quadrotor UAV. By using the image features in the virtual image plane, a velocity controller is derived (TMECH 2017). One of the biggest challenges in IBVS is to on-line estimate the depth. We propose a nonlinear observer to simultaneously estimate the depth of the point features and the velocity of the quadrotor using visual feedback. Experimental tests, including the comparison with an extended-Kalman-filter based observer, are conducted to verify the validity of the observer (TMECH 2018). The visibility problem may lead to an failure in vision servoing for UAVs. To guarantee the visibility, we define a visibility constraint based on control barrier function. The control inputs are minimally modified to satisfy the visibility constraint, thus preserving visibility (TMECH 2019).
hpp Visual Tracking of UAVs
Image-based visual tracking (IBVT) problem of quadrotors is challenging. Because for such systems, the relationship between the control inputs and the image feature's motion is often complex. We propose a nonlinear controller using features defined in virtual plane for the quadrotor to track a moving target. The target is assumed to be moving with unknown, time varying and bounded linear velocity, linear acceleration, and angular velocity and acceleration of the yaw angle. The controller is proved uniformly ultimately bounded (UUB) by means of Lyapunov analysis (ASCC 2017). By adopting the virtual camera approach and choosing image moments, we design the trajectories of the image features in image space to perform the image-based visual control task of the quadrotor. A feature's trajectory tracking controller is proposed to track the designed trajectories. The stability of the proposed tracking controller is analyzed and proved by means of Lyapunov analysis (TIE 2018).
hpp Real‐time UAV Trajectory Planning
The trajectory planning algorithm is the core of autonomous navigation, which can undoubtedly greatly enhance the safety of flight. Due to the necessity of planning safe and dynamically feasible trajectories for quadrotors in unknown environments, we proposed a trajectory planning framework based on B‐spline and kinodynamic search. This framework can be used for a limited‐sensing quadrotor, and the flight is safe and effective along with these trajectories. First, a B‐spline based nonuniform kinodynamic (BNUK) search algorithm is proposed to generate dynamically feasible trajectories efficiently. The characteristics of nonuniform search make the generated trajectories safe and reasonable time‐allocation. Then, a trajectory optimization method based on control point optimization is proposed. Multiple outdoor flight experiments show the effectiveness of the proposed framework. (Journal of Field Robotics 2020)
hpp Contact-based aerial manipulation
Contact-based aerial interaction control in positioning system denied environments is a challenging issue. An image-based impedance control strategy for force tracking of an unmanned aerial manipulator is proposed (TASE 2022). To achieve force tracking under the visual guidance, we design an adaptive visual impedance control method which adjusts the target stiffness according to the force tracking error and the visual feature error. The closed-loop system is proved asymptotically stable by means of Lyapunov analysis. Besides, we propose a vision-guided approach for the impedance control of an aerial manipulator based on line features, with the goal of physical interaction with unknown environments. To this end, a nonlinear observer is proposed to online estimate the 3-D parameters of the environment. These parameters are adopted to estimate the interaction matrix related to the image features. By planning the image-space trajectory and the distance, desired interaction behavior can be uniquely specified without relying on any Cartesian information of the system. (TASE 2022)

Medical Robot

For medical robots, our research is mainly focused on the optimization of the specified operations, and is committed to improve the level of automation in the operation. Specifically, in the perception part, we study the optimization of the 3D tooth segmentation; in the control part, we develop the automatic manipulation of soft tissue in robot-assisted surgery (RAS), including deformation trajectory control, cutting control, etc.

hpp 3D Tooth Segmentation
Due to the variability and complexity of geometric feature distribution on dental meshes, traditional segmentation methods based on geometry often fail. We improve the region growing algorithm, with multiple parameters jointly evaluating the region similarity, in order to enhance the adaptability of the algorithm to the actual application scene requirements. Besides, we design a parameter adaptive method to raise efficiency and provide a multi-level label optimization algorithm for segmentation refinement (RCAR 2021). To improve the labeling accuracy and robustness against some tough conditions including tooth crowding, we establish a large-scale 3D dental mesh data set and propose a deep neural network called VFENet for 3D tooth segmentation and labeling (TMI submission).
hpp Automatic Cutting Control of Deformable Objects
Automatic cutting is an essential task in the field of robot-assisted surgery (RAS). The high-dimensional deformation and time-varying topology caused by the interaction between the object and the cutting tool make relevant research preliminary. Thus, a cutting control method based on vision and force feedback is proposed to cut along a pre-designed trajectory automatically. Through force feedback, the resistance of the cutting tool can be reduced by adopting the pressing and slicing approach (WCICA 2018). To achieve more precise automatic cutting control to adapt to RAS, we developed an automatic cutting control algorithm for deformable objects based on surface tracking. A dynamic controller based on the combined features is designed. Compared with the cutting controller based on point features, this method can prevent the failure of the servo task caused by partial occlusion and invisible feature points (TMECH 2020).
hpp Visual Tracking Control of Deformable Objects
Automatic manipulation of deformable objects is a challenging problem. Improper operations, such as excessive stretching, collision, are easy to cause damages to the deformable objects. Thus, not only the final configuration but also the trajectory of the deformation is supposed to be controlled during the process of interaction. In this paper, a model-free method to control the trajectory of the deformation in the unknown environment is proposed. We design an adaptive dynamic controller that adaptively estimates the deformation Jacobian matrix (DJM) online based on function approximation techniques (FAT), which approximates nonlinear functions with an arbitrary small error, avoiding modeling for compliant objects. Besides, we introduce a virtual force to improve the manipulability of the method. The stability of the proposed adaptive algorithm and the boundedness of internal signals are proved by Lyapunov theory whenever the approximation error is nonnegligible or negligible. Experiment results validate the efficiency of the algorithm proposed. (TIE 2021)
hpp Soft Surgical Robot
The soft surgical robot system for cardiothoracic surgery has been independently developed, using flexible materials to cast prototypes to further ensure the safe interaction with the heart and lungs and other important organs during the operation. According to the needs of surgery, the medical image feedback is integrated, and the doctor can realize the man-machine interactive movement strategy based on the telecontrol lever, and realize the movement of the robot in the cavity such as forward and steering. The prototype has verified its operational performance in the organ model, and successfully carried out 5 live animal surgery experiments to test and verify the actual operating effect of the snake-shaped surgical robot. (Surgical Endoscopy and other Interventional Techniques, 2016)

Reinforcement-learning Control

Our group is conducting various reinforcement-learning based researches on the topic of robot and vehicle control, mainly including the reinforcement-learning based multi-robot visual navigation and coordination, reinforcement-learning enhanced robot navigation in complex environments, mobile manipulator control based on reinforcement-learning, and efficient robotic task learning with base-controller-augmented algorithms.

hpp Reinforcement-learning based Multi-robot Visual Navigation
In the unknown, map-free dynamic environment, robots should combine their own sensor information with the group messages transmitted through a decentralized information exchange mechanism to achieve effective and stable navigation and obstacle avoidance strategies. Without relying on global maps, nor the precise positioning and instant guidance, we hope that first-person-view data from sensors such as camera and laser is directly used as the robot's perception input, the graph neural networks (GNNs) are used to realize distributed information interaction between robots in a certain range, and finally achieve a wider range of information perception and more efficient scheduling and coordination strategies. In this process, we will flexibly use deep reinforcement learning, auxiliary tasks, curriculum learning and related methods to solve a series of scientific and engineering problems and improve the perception, scheduling, coordination and decision-making capabilities of multi-robot navigation systems.
hpp Reinforcement-learning Navigation in Complex Environments
In an unknown environment with static obstacle and crowd, robot only uses local information collected by its own sensors to realize social-aware, efficient and safe navigation as well as obstacle avoidance. We use information within a certain range around the robot and surrounding pedestrians in the local observation as input. Focusing on understanding the behavioral interactions between pedestrians and robots, pedestrians and pedestrians through the graph neural network(GNNs), so as to accurately predict the trajectory of pedestrians and help the robot make more forward-looking behavioral output. In this process, we use deep-reinforcement-learning, social-awareness guidance mechanism, imitation-learning and other methods, so that making the robot's navigation process follow social rules and cause less impact on human behavior.
hpp Mobile Manipulator Control Based on Reinforcement-learning
Mobile manipulator, capable of automatically working alongside humans and performing tasks in an unstructured environment, is becoming an increasingly important focus of robotics research. However, mobile manipulation is usually more challenging than fixed-base manipulation due to the varying manipulation dynamics and uncertain external perturbations. Reinforcement learning (RL) methods have been demonstrated to be capable of learning continuous robot controllers from interactions with the environment, which shows the ability to handle variability and uncertainty. We hope to achieve the whole body non-planar surface wiping task using image, force, and arm joint information without pre-designed computational models. Hierarchical plannings have been considered in the following strategy: (1) high-level RL which learns the phase-type and sets a subgoal; (2) online redundancy resolution based on the neural-dynamic optimization algorithm in operational space; and (3) low-level RL in joint space. At this level, the dynamic movement primitives (DMPs) have been considered to model and learn the joint trajectories, and then the RL is employed to learn the trajectories with uncertainties.
hpp Efficient Robotic Task Learning with Base-controller-augmented Algorithms
Application of Deep-reinforcement-learning algorithms in real-world robotic tasks faces many challenges. On the one hand, reward-shaping for complex tasks is difficult and may result in sub-optimal performances. On the other hand, a sparse-reward setting renders exploration inefficient, and exploration using physical robots is of high-cost and unsafe. Hence we wish to accelerate learning utilizing existing controllers. Algorithms like these could incorporate the base controllers into stages of exploration, value estimation or policy updates, and should be able to learn state-based or image-based policies that steadily outperform the base controllers. Compared to learning from demonstrations, these algorithms could improve sample efficiency by orders of magnitude and learn online in a safe manner, thus it can bear the potential of leveraging existing industrial robot manipulation systems to build more flexible and intelligent controllers.

Multi-robot Control and Large-scale Scheduling

For multi-robot systems, our group mainly focuses on two aspects: efficient planning and coordination for large-scale robotic systems, and robust formation control for mobile robots. The former one aims to the task assignment, path planning and local motion coordination problems for thousands of robots in the presence of uncertainties; the latter one aims to the synchronous formation control of mobile robots considering communication constraints and robot failures.

hpp Robust Formation Control of Multiple Robots
Firstly, in terms of vision-based multi-robot formation control: in the absence of global GPS positioning system, we combine the measurement information of the vision camera, odometer and attitude and heading reference system, and proposes an adaptive algorithm to estimate the relative position between the leader and the follower. On this basis, the corresponding formation control algorithm is designed using the estimated relative position information. Secondly, in the aspect of multi-robot formation control in the networked communication environment: in response to the communication delay and information sampling problems in the communication network, a distributed synchronous formation controller based on sampled-data is designed, and the Lyapunov-Krasovsky method provides sufficient conditions to ensure the exponential convergence of the control system. At the same time, in order to solve the problem of the decline of the movement synchronization caused by robot failures, a distributed formation self-repairing algorithm based on the topology switching method and the local negotiation mechanism is proposed. (TSMC 2020, TSMC 2020, ICRA 2018, ChiCC 2017)
hpp Multi-robot Transportation Planning with Incidental Delivery Behaviors
Aiming for the practical applications of warehousing logistics and social services, we study the multiple mobile robots cooperative transportation problem. A group of transportation robots are assigned to accomplish a set of pair-wised transportation tasks, where each task has different time window constraints and capacity requirements. We incorporate the auction mechanism and propose an incidental delivery based approach to solve this problem. The proposed approach is fully distributed and can be extended to fulfill the online transportation planning or re-planning requirement in the dynamic environment. Simulation results demonstrate the advanced performance of the proposed approach in large-scale transportation problems, and experiment results validate the practical applicability of the proposed approach in practical warehousing applications. In particular, we demonstrate that how incidental delivery behaviors improve the efficiency of multi-robot transportation systems. (RCAR 2017)
hpp Scheduling and Coordination of Large-scale Robotic Networks
This research focuses on the problem of task allocation, cooperative path planning and motion coordination of large-scale transportation system with thousands of robots, aiming for practical applications in autonomous driving system and robotic warehouses. We are interested in resolving the lifelong planning problem of large-scale robot system and guarantee the coordination performance in the presence of robot motion uncertainties and communication failures (i.e., avoid traffic jams and robot deadlocks). Currently, a hierarchical system is presented. In high level, integrated task allocation and path planning is achieved by introducing a conflict graph based optimization approach. In road level, receding horizon planning is implemented to solve problems caused by motion and communication uncertainties. The proposed approach has been validated by simulations with more than two thousand robots and laboratory experiments with several mobile robots. This work won the Best Conference Paper Award of 2020 IEEE International Conference on Real-time Computing and Robotics. (TASE 2020, RCAR 2020 (Best Paper Award), TASE submission)

Machine Vision Projects

For machine vision projects, we focus on solving localization, mapping and perception problems with low-cost cameras. Our algorithm is dedicated to solving the challenges of limited computing resources, environmental polytrope, sensor degradation and so on in practical applications such as indoor sweeping robots, indoor humanoid robots and scenic area monitoring systems. Our work includes stereo-based simultaneous localization and mapping, stereo-based obstacle detection and rgbd-based dense 3D mapping.

hpp IMU, Odometer and Stereo-fused Localization and Mapping
The limited computing resources of the embedded platform, the variable indoor lighting environment and the lack of features bring challenges to the real-time, robust and accurate localization and mapping of indoor mobile robots. In order to improve the adaptability to darkness and texture loss, an IMU, odometer and stereo fused slam framework based on graph-based optimization are proposed. Line features are added to make up for the lack of point features and provide spatial geometry information of Manhattan space. Binocular stereo matching, error model based keypoints selection and hash tree based keyframe management are added to improve operating efficiency. Our framework is proved to function well in extreme environments such as darkness, bright light and lack of texture.
hpp Stereo-based Obstacle Detection
Obstacles with varying shapes, sizes and types bring challenges to obstacle detection. We propose a 2D-detection driven 3D obstacle detection algorithm with a stereo camera. Mature deep-network based 2D detection methods such as yolo or faster-rcnn can directly utilized to provide reliable 2D detection results. At the same time, a mask of obstacle with disparity map can be obtained quickly with the prior of camera height. With the result of 2D detection and obstacle mask, point cloud of the overlapping part of the 2D detection box and obstacle mask is recovered. Object cluster is then selected from multiple point cloud clusters based on the voting mechanism considering distance to the camera, the number of points and IOU (Intersection over Union). Our framework is proved to detect objects with small sizes such as mahjong and badminton and objects with variable shapes such as dogshit and wire.
hpp RGBD-based 3D Reconstruction
Dense modeling of real-time local environments with low-cost rgbd camera remains a challenging problem. Limited computing resources of the mobile platform, texture lackness, and blur problem caused by fast camera movement bring it difficulty for mobile robots to obtain a real-time dense perception of the local and global environment with high quality. We propose a 3D dense mapping algorithm for the local and global environment based on global 3D reconstruction. The adaptability of localization in low-texture plane environment is improved by providing ICP with initial value from RGBD localization method. Calculation efficiency is improved by device-host swap mechanism and point cloud regularization. Our framework is proved to obtain real-time dense mapping of local environment and global-consistent environment of high quality.