Mingyuan Zhou (University of Texas at Austin)

The Poisson gamma belief network

A key issue in deep learning is to define an appropriate network structure, including both the depth of the network and the width of each hidden layer, which may be naturally addressed with completely random measures. We propose the Poisson gamma belief network (PGBN), which factorizes each of its layers into the product of a connection weight matrix and the nonnegative real hidden units of the next layer, to infer a multilayer representation of high-dimensional count vectors. We use the
gamma-negative binomial process together with a layer-wise training strategy to infer the network structure, allowing the width of each hidden layer to grow without bound. We further demonstrate that with a fixed budget on the width of the first layer, the PGBN can increase its number of hidden layers to boost its performance. We propose an efficient upward-downward Gibbs sampler to jointly train all the hidden layers of the PGBN, with example results on text analysis illustrating its efficacy and unique properties.