There is an unavoidable time offset between the camera stream and the inertial measurement unit (IMU) data due to the sensor triggering and transmission delays, which will seriously affect the accuracy of visual-inertial odometry (VIO). A novel online time calibration framework via double-stage EKF for VIO is proposed in this paper. First, the first-stage complementary Kalman filter is constructed by adapting the complementary characteristics between the accelerometer and the gyroscope in the IMU, where the rotation result predicted by the gyroscope is corrected through the measurement of the accelerometer so that the IMU can output a more accurate initial pose. Second, the unknown time offset is added to the state vector of the VIO system. The estimated pose of IMU is used as the prediction information, and the reprojection error of multiple cameras on the same feature point is used as the constraint information. During the operation of the VIO system, the time offset is continuously calculated and superimposed on the IMU timestamp to obtain the data synchronized by the IMU and the camera. The Schur complement model is used to marginalize the camera state that carries less information in the system state, avoiding the loss of prior information between images, and improving the accuracy of camera pose estimation. Finally, the effectiveness of proposed algorithm is verified using the EuRoC dataset and the real experimental data.

We present and discuss a runtime architecture that integrates sensorial data and classifiers with a logic-based decision-making system in the context of an e-Health system for the rehabilitation of children with neuromotor disorders. In this application, children perform a rehabilitation task in the form of games. The main aim of the system is to derive a set of parameters the child’s current level of cognitive and behavioral performance (e.g., engagement, attention, task accuracy) from the available sensors and classifiers (e.g., eye trackers, motion sensors, emotion recognition techniques) and take decisions accordingly. These decisions are typically aimed at improving the child’s performance by triggering appropriate re-engagement stimuli when their attention is low, by changing the game or making it more difficult when the child is losing interest in the task as it is too easy. Alongside state-of-the-art techniques for emotion recognition and head pose estimation, we use a runtime variant of a probabilistic and epistemic logic programming dialect of the Event Calculus, known as the Epistemic Probabilistic Event Calculus. In particular, the probabilistic component of this symbolic framework allows for a natural interface with the machine learning techniques. We overview the architecture and its components, and show some of its characteristics through a discussion of a running example and experiments.

Localization based on visual natural landmarks is one of the state-of-the-art localization methods for automated vehicles that is, however, limited in fast motion and low-texture environments, which can lead to failure. This paper proposes an approach to solve these limitations with an extended Kalman filter (EKF) based on a state estimation algorithm that fuses information from a low-cost MEMS Inertial Measurement Unit and a Time-of-Flight camera. We demonstrate our results in an indoor environment. We show that the proposed approach does not require any global reflective landmark for localization and is fast, accurate, and easy to use with mobile robots.

The design and experimental works of an autonomous robotic platform for road marks painting are presented in this paper as the first autonomous system of its kind. The whole system involves two main sub-systems, namely: an autonomous mobile robot navigation system which is used for recognizing the roads and estimating the position of road marks, and automatic road marks painting system that is attached to the mobile robot platform to control the spray of the paint on the road’s surface. The experimental results show the capability of the proposed system to perform the task of autonomous road marks painting with accuracy of ±10 cm.

Estimating the robot state within a known map is an essential problem for mobile robot; it is also referred to “localization”. Even LiDAR-based localization is practical in many applications, it is difficult to achieve global localization with LiDAR only for its low-dimension feedback, especially in environments with repetitive geometric features. A sensor-fusion-based localization system is introduced in this paper, which has the capability of addressing the global localization problem. Both LiDAR and vision sensors are integrated, making use of the rich information introduced by vision sensor and the robustness from LiDAR. A hybrid grid-map is built for global localization, and a visual global descriptor is applied to speed up the localization convergence, combined with a pose refining pipeline for improving the localization accuracy. Also, a trigger mechanism is introduced to solve kidnapped problem and verify the relocalization result. The experiments under different conditions are designed to evaluate the performance of the proposed approach, as well as a comparison with the existing localization systems. According to the experimental results, our system is able to solve the global localization problem, and the sensor-fusion mechanism in our system has an improved performance.

A sensor fusion method was developed in order to track crop rows, considering various vegetation levels, for various crops. This application consisted to use a laser sensor, an inertial measurement unit and a color camera, in a fusion mode, to get a set of points corresponding to crop rows and eliminate noise like grass or leaves in environment, in real time. After applying a method such as Hough or Least Square (LS) technique for obtaining the geometric data of the crop line, automatic control operations were applied to realize the crop row tracking operation, with the desired lateral deviation parameter, taking into account the robot angular deviation and the temporal aspect, to realize the task with accuracy and without oscillations. The results showed the robustness of fusion method, to get a stable autonomous navigation for crop row tracking, particularly in the vineyards, with many perturbations such as bumps, hole and mud, and speeds between 1 and 2 m s-1. The mean lateral error between desired and obtained trajectory varied between 0.10 and 0.40 m, depending of speed and perturbations.

This paper proposes a Radio Frequency Identification (RFID)/ in-vehicle sensors fusion strategy for vehicle positioning in completely Global Positioning System (GPS)-denied environments such as tunnels. The strategy employs a two-step approach, namely, the calculation of the distances between the RFID tags and the reader, and then the global fusion estimation of vehicle position. First, a Least Square Support Vector Machine (LSSVM) algorithm is developed to obtain distance. Next a novel Federated Unscented Kalman Filter (FUKF) is designed to realise the global fusion. The decentralised federated filter is adopted to combine the data from RFID and in-vehicle sensors, and the UKF is employed to design a local filter since it has better ability to deal with a nonlinear problem than an Extended Kalman Filter (EKF). Due to the optimised layout of RFID tags and the application of the decentralised filter, the number of tags is reduced. Finally, the feasibility and effectiveness of the proposed strategy are evaluated through experiments.

Recently, small-sized compact electric vehicles have been in demand for short-distance travel in urban areas, although battery charging in electric vehicles present in the market is still problematic. Borrowing from the concept of a mobile inverted pendulum system, in this paper, a two-wheel robotic vehicle system is implemented and controlled as the future personal transportation device called the TransBOT. The TransBOT has two driving modes: a regular vehicle mode, where stable contact on the ground is maintained by two wheels and two casters, and the balancing mode, which maintains the stable posture with two wheels on the ground. The two-wheel balancing mechanism can be used as a transportation vehicle in narrow and busy urban areas. Gain scheduling control methods based on linear controllers are used for different drivers. In addition, desired balancing angles are specified for the different sizes of drivers in order to have a stable balancing control performance. These desired balancing angle values have been found by empirical studies. Experimental studies with drivers of different weights, as well as indoor and outdoor driving tasks, were conducted to ensure the feasibility of TransBOT.

The paper examines the problem of dynamic ship positioning with the use of Kalman Filter- and Particle Filter-based sensor fusion algorithms. The proposed approach enables to estimate accurately the ship's state vector by fusing the vessel's position and heading measurements coming from on-board sensors together with distance measurements coming from sensors located at the coast (e.g. radar). The estimated state vector is used in turn, in a control loop, to regulate the horizontal position and heading of the vessel. The performance of dynamic positioning of the ship based on Kalman and Particle Filtering is evaluated through simulation experiments.

This paper describes a landmark detection and localization using an integrated laser-camera sensor. Laser range finder can be used to detect landmarks that are direction invariant in the laser data such as protruding edges in walls, edges of tables, and chairs. When such features are unavailable, the dependant processes will fail to function. However, in many instances, larger number of landmarks can be detected using computer vision. In the proposed method, camera is used to detect landmarks while the location of the landmark is measured by the laser range finder using laser-camera calibration information. Thus, the proposed method exploits the beneficial aspects of each sensor to overcome the disadvantages of the other sensor. While highlighting the drawbacks and limitations of single sensor based methods, an experimental results and important statistics are provided for the verification of the affectiveness sensor fusion method using Extended Kalman Filter (EKF) based simultaneous localization and mapping (SLAM) as an example application.

This paper describes some key points of a new tool being currently developed by Eurocontrol for the assessment of air traffic control (ATC) multisensor trackers performance. It summarizes the algorithmic foundations of the high-accuracy trajectory reconstruction process used to obtain reference trajectories from recorded measures. These trajectories will serve as a reference for the evaluation of the accuracy of ATC data processing centers. The performance of the system is illustrated with some reconstruction experiments on synthetic and real data.

This paper argues the case for extracting as complete a set of sensory data as practicable from scenes consisting of complex assemblages of objects with the goal of completing the task of scene analysis, including placement, pose, identity and relationship amongst the components in a robust manner which supports goal directed robotic action, including collision-free trajectory planning, grip site location and manipulation of selected object classes.

The emphasis of the paper is that of sensor fusion of range and surface colour data including preliminary results in proximity, surface normal directionality and colour based scene segmentation through semantic-free clustering processes. The larger context is that of imbedding the results of such analysis in a graphics world containing an articulated robotic manipulator and of carrying out experiments in that world prior to replication of safe manipulation sequences in the real world.

The Kalman filter (KF) is a form of optimal estimator characterized by recursive evaluation, which has been widely applied to the navigation sensor fusion. Utilizing the KF requires that all the plant dynamics and noise processes are completely known, and the noise process is zero mean white noise. If the theoretical behaviour of the filter and its actual behaviour do not agree, divergence problems tend to occur. The adaptive algorithm has been one of the approaches to prevent divergence problems in the Kalman filter when precise knowledge on the system models is not available. Two popular types of adaptive Kalman filter are the innovation-based adaptive estimation (IAE) approach and the adaptive fading Kalman filter (AFKF) approach. In this paper, an approach involving the concept of the two methods is presented. The proposed method is a synergy of the IAE and AFKF approaches. The ratio of the actual innovation covariance based on the sampled sequence to the theoretical innovation covariance will be employed for dynamically tuning two filter parameters – fading factors and measurement noise scaling factors. The method has the merits of good computational efficiency and numerical stability. The matrices in the KF loop are able to remain positive definitive. Navigation sensor fusion using the proposed scheme will be demonstrated. Performance of the proposed scheme on the loosely coupled GPS/INS navigation applications will be discussed.

An algorithm for three-level hierarchical sensor fusions has beenproposed and applied to environment map building with enhanced accuracy and efficiency. The algorithm was realized through the two new types of sensor modules, which are composed of a halogen lamp-based active vision sensor and a semicircular ultrasonic (US) and infrared (IR) sensor system. In the first-level fusion, the US and IR sensor information is utilized in terms of the geometric characteristics of the sensor location. In the second-level fusion, the outputs from the US and IR sensors are combined with the sheet of halogen light through a proposed rule base. In the third-level fusion, local maps from the first- and second-level fusion are updated in a probabilistic way for a very accurate environment local map. A practical implementation has been carried out to demonstrate the efficiency and accuracy of the proposed hierarchical sensor fusion algorithm in environment map building.

In the work package “Integrated Positioning” of the research project NAVIO (Pedestrian Navigation Systems in Combined Indoor/Outdoor Environments) we are dealing with the navigation and guidance of visitors to our University. The start points are the public transport stops in the surroundings of the Vienna University of Technology and the system users should be guided to certain office rooms or persons. For the user's position determination different location sensors are employed, i.e., for outdoor positioning GPS and dead reckoning sensors, such as a digital compass and gyro for heading determination, accelerometers for the determination of the distance travelled, a barometric pressure sensor for altitude determination and, for indoor areas, location determination using WiFi fingerprinting. All sensors and positioning methods are combined and integrated using a Kalman filter. An optimal estimate of the current location of the user is obtained using the filter. To perform an adequate weighting of the senors in the stochastic filter model, the sensor characteristics and performance were investigated in several tests. The tests were performed in different environments either with free satellite visibility, in urban canyons or inside buildings. The tests have shown that it is possible to determine the user's location continuously with the required precision and that the selected sensors provide a good performance and high reliability. Selected tests results and our approach are presented in the paper.

Low cost automation often requires accurate positioning. This happens whenever a vehicle or robotic manipulator is used to move materials, parts or minerals on the factory floor or outdoors. In last few years, such vehicles and devices are mostly autonomous. This paper presents the method of sensor fusion based on the Adaptive Fuzzy Kalman Filtering. This method has been applied to fuse position signals from the Global Positioning System (GPS) and Inertial Navigation System (INS) for the autonomous mobile vehicles. The presented method has been validated in 3-D environment and is of particular importance for guidance, navigation, and control of mobile, autonomous vehicles. The Extended Kalman Filter (EKF) and the noise characteristic have been modified using the Fuzzy Logic Adaptive System and compared with the performance of regular EKF. It has been demonstrated that the Fuzzy Adaptive Kalman Filter gives better results (more accurate) than the EKF. The presented method is suitable for real-time control and is relatively inexpensive. Also, it applies to fusion process with sensors different than INS or GPS.

In equipment monitoring and diagnostics, it is very important to distinguish between a sensor failure and a system failure. In this paper, we develop a comprehensive methodology based on a hybrid system of AI and statistical techniques. The methodology is designed for monitoring complex equipment systems, which validates the sensor data, associates a degree of validity with each measurement, isolates faulty sensors, estimates the actual values despite faulty measurements, and detects incipient sensor failures. The methodology consists of four steps: redundancy creation, state prediction, sensor measurement validation and fusion, and fault detection through residue change detection. Through these four steps we use the information that can be obtained by looking at: information from a sensor individually, information from the sensor as part of a group of sensors, and the immediate history of the process that is being monitored. The advantage of this methodology is that it can detect multiple sensor failures, both abrupt as well as incipient. It can also detect subtle sensor failures such as drift in calibration and degradation of the sensor. The four-step methodology is applied to data from a gas turbine power plant.