We use cookies.
By using the site, you agree to our Privacy Policy.

Laboratory Tensor Networks and Deep Learning for Applications in Data Mining

Invited researcher Andrzej Stanislaw Cichocki
Contract number
14.756.31.0001
Time span of the project
2017-2021

As of 30.01.2020

33
Number of staff members
32
scientific publications
General information

Name of the project:    Tensor networks and deep learning for intelligent data analysis

Strategy for Scientific and Technological Development Priority Level: а


Goals and objectives

Research directions: Development of algorithms of tensor decompositions and neural networks for data compression and machine learning, as well as practical application of neural networks and tensor decompositions in biomedicine, telecommunications, agrochemistry, robot control, quantum computers and other domains

Project objective: Computer science and information technologies


The practical value of the study

  • We have reviewed current challenges in tensorized machine learning algorithms and found a found a number of new solutions of the problem of tensorization and distributed representation of structured data. It has been proven that tensor networks give an opportunity to perform efficient distributed computations for super-big volumes of data thus reducing or even eliminating the effect of «curse of dimensionality». Tensorized algorithms have been developed for a number of applied problems.
  • The Laboratory has studied possible methods of practical usage of low-dimensional tensor approximations for solving a wide range of tasks of large-scale polylinear dimensionality reduction and related problems of optimization on tensor manifolds that are very hard to solve using classical machine learning methods.
  • Our researchers have studied methods of training of generative models that uses approximate Bayesian inference. A method of training variational autoencoder has been proposed that is using K-selective evaluations on sustainable credibility. The method is based on replacement of the target functional of the variational autoencoder, marginal credibility with its modification that is resistive to noise. Also for efficient maximization of noise-prone credibility credibility we have proposed and researched a number of lower estimates generalizing weighted K-selective evaluations.
  • We have developed a model based on artificial neural networks (including recurrent networks) for optimization of functionals in multi-dimensional spaces and proposed a method of training that significantly reduces the effect of excessive training. Additionally, we have developed a probabilistic model of group thinning and proven its applicability to modern deep architectures of computer vision of types VGG and ResNet. Results of numerical experiments have demonstrated high level of compression and acceleration without loss of forecasting capability in case of usage of the proposed model.
  • We have created a database marked X-ray images that allows to test machine learning algorithms for segmentation, noise removal, automatic detection of location of objects in images and intelligent classification of data. This database can be used for development of automated decision-making support systems for medical applications.
  • We have implemented a new layer that uses classic computer vision methods for various architectures of convolutional artificial neural networks allowing to significantly speed up the process of automatic segmentation of images and searching for objects in images while maintaining quality of results, which makes this approach a promising one for usage in real-time systems, including those running on power-restricted mobile platforms. The proposed model of an artificial neural network based on the new layer is not inferior to ENet, one of the fastest modern models, in terms of quality.
  • The Laboratory has created new models based on deep artificial neural networks for problems of photorealistic synthesis of images including such directions as changing attributes of the image, faces of people, neural network transfer of style through the example of generation of items of clothing, semantic segmentation of items of clothing and alpha-matting of hair, summation of textures and biological images
  • Using the mechanisms of generalized tensor decomposition we have demonstrated universality and existence of efficiency of depth in recurrent neural networks. We have also proposed a predictor modeling all the two-dimensional interactions of multi-dimensional data by representation of an exponentially big tensor of parameters in the compact TT-format, a stochastic teaching algorithm based on Raman optimization has been created.
  • We have found theoretical results for local linear convergence of alternating optimization algorithms for multilinear and low-rank optimization, in particular, for the alternating least squares algorithm that does depend on representation of low-rank tensors.
  • A new method has been developed for studying hidden Markov chains. The method uses tensor networks. An efficient iterative approach to compression and acceleration of neural networks has been proposed. It allows to reach a significant degree of compression preserving forecasting capabilities of the model.
  • An algorithm has been proposed for building biomarkers of medical plants using machine learning methods with tensor decompositions based on chromato-mass-spectrometry data om the problem species identification. We also developed a method of building embedded graphs that can be used in any machine learning algorithm as a feature vector describing the graph. Additionally we proposed and researched non smooth non negative factorization and shown its connection to a special type of deep neural systems and its efficiency in applications to cluster analysis.
  • The Laboratory has conducted experimental research of qualities of loss functions of deep neural networks and shown that local optima in weights space obtained in practice are not isolated and can be connected with curves along which loss functions and classification error maintain their values.
  • We have proposed a new method of building sets of incongruous near optimal configurations of generative artificial neural networks since for variational approximation storage of K independent generative networks is costly in terms of memory, an efficient ensembling method has been proposed that requires storage of just three artificial neural networks.
  • It has been shown that to restore a high resolution image from an image of image with lower resolution using a neural network it is namely the neural network's architecture that is important, not the amount o data used for training its parameters. We have proposed trainable latent convolutional manifolds which is unique to every image of the training dataset that of a convolutional neural network, as well as a generator that is common for all the images which is a convolutional neural network as well and is optimized during training with latent convolutional manifolds which allows to find the optimal latent representation for any dataset and to efficiently remove any blur and distortion in images.
  • Our researchers have proposed a new architecture for neural networks that allows to recognize objects in images with high precision by using a lightweight convolutional network that can be run on low capability devices mobile phones not requiring utilization of a GPU.
  • A new neural network architecture has been developed to solve the problem for producing a dense representation of depth maps from 3D data from laser distance gauges. The proposed model extrudes local nuclei from the corresponding auxiliary RGB image, local nuclei evaluate directions of diffusion for each pixel which allows to propagate information through the proposed architecture in accordance with the whole scene and reach precision exceeding that of modern published methods.

Implemented results of research:

  • Obtained results will become the scientific and technological base for creation of software products capable of working on mobile platforms using the created technology of tensorization of deep neural networks. This will make offline processing of information possible – which can be used for current problems of machine learning and data analysis on mobile devices without contacting a server.
  • The implemented neural network layer for automatic segmentation of images and search for objects in images have been submitted to the PyTorch open machine learning library.
  • The Laboratory has filed an application for state registration of a software program «A program for creating recommendation systems «Polara» that is a result of intellectual work of the Laboratory. The program is designed for quick and user-friendly creation of new recommendation algorithms as well as for comprehensive analysis of quality of their work.

Education and career development:

  • The Laboratory has created 5 education courses for masters: «Deep Learning», «Numerical Linear Algebra», «Bayesian Methods of Machine Learning», «Convex Optimization and Applications», «Machine Learning and Applications». 
  • We have prepared 2 volumes of the monograph in tensor methods «Tensor networks for dimensionality reduction and large scale optimization».
  • We have organized 2 seminars: «Numerical tensor methods and machine learning», «Journal Club» (in machine learning and tensor decompositions).
  • One candidate dissertation has been defended.
  • 3 students have been admitted to the postgraduate school.

Collaborations:

  • Huawei (China PR): research of convolutional networks
  • LG Electronics (South Korea): research of methods of fast tensor convolutions of networks

Hide Show full
Zhang Y., Wang Y., Zhou G., Jin J., Wang B., Wang X., & Cichocki A.
Multi-kernel extreme learning machine for EEG classification in brain-computer interfaces. Expert Systems with Applications 96: 302-310 (2018).
Xu X., Wu Q., Wang S., Liu J., Sun J., & Cichocki A.
Whole Brain fMRI Pattern Analysis Based on Tensor Neural Network. IEEE Access 6: 29297-29305 (2018).
Khrulkov V., Oseledets I.
Desingularization of bounded-rank matrix sets. SIAM Journal on Matrix Analysis and Applications 39(1): 451-471 (2018).
Sole-Casals J., Caiafa C. F., Zhao Q., Cichocki A.
Brain-Computer Interface with corrupted EEG data: A Tensor Completion Approach. Cognitive Computation 10(6): 1062-1074 (2018).
Kharyuk P., Nazarenko D., Oseledets I., Rodin I., Shpigun O., Tsitsilin A. & Lavrentyev M.
Employing fingerprinting of medicinal plants by means of LC-MS and machine learning for species identification task. Scientific reports 8(1): 17053. (2018).
Other laboratories and scientists
Hosting organization
Field of studies
City
Invited researcher
Time span of the project
Computer Synthesis of Chemical Compounds

  G. A. Krestov Institute of Solution Chemistry of the Russian Academy of Sciences

Computer science and information technologies

Ivanovo

Tetko Igor

2021-2023

Multi-scale Neurodynamics for Smart Systems

Skolkovo Institute of Science and Tehnology

Computer science and information technologies

Moscow

Wang Jun

2021-2023

Laboratory of Non-linear and Microwave Photonics

Ulyanovsk State University

Computer science and information technologies

Ulyanovsk

Taylor James Roy

United Kingdom

2021-2023