Designing Information-Rich Embedding Spaces

Ever since Mikolov et al introduced Word2Vec in 2013, there has been a desire to make models which map data into an information-rich embedding spaces. However, with noisy data or data from various channels, the learned embedding space often captures information irrelevant to the latent concepts we are trying to model (whether that be semantic content, speaker information, etc.).  For the purpose of latent modelling, this extra information can be considered noise.  We explore the use of regularization techniques to limit the amount of information encoded in the embedding.  When combined with loss terms that prioritizes useful information, we have been able to learn embedding spaces which correlate better to the latent variables we are trying to model.  Moreover, these embedding spaces can be learned in a weakly supervised fashion where the only supervisory signal comes in the form of paired sensory inputs.