An efficient point-set registration algorithm with dual terms based on total least squares

•The proposed TLS-based criterion includes two dual terms which are only governed by the single row-orthonormal STM.•The rationality of the TLS-based criterion is explained.•We formulate the soft-decision making criterion which includes two outlier parameters and the deterministic annealing technology.•We improve the accuracy of correspondence estimation and enhance the suppression of bilateral outliers by using two dual terms.