Файл:3048-greedy-layer-wise-training-of-deep-networks.pdf

YoshuaBengio,PascalLamblin,DanPopovici,HugoLarochelle Universite´deMontre´al Montre´al,Que´bec {bengioy,lamblinp,popovicd,larocheh}@iro.umontreal.ca

Abstract

Complexity theory of circuits strongly suggests that deep architecture scan be much more efficient (sometimes exponentially) than shallow architectures, in terms of computational elements required to represent some functions. Deep multi-layer neural networks have many levels of non-linearities allowing them to compactly represent highly non-linear and highly-varyingfunctions. However, until recently it was not clear how to train such deep networks,since gradient-based optimization starting from randominitialization appears to often get stuck in poor solutions. Hintonet al. recently introduced a greedy layer-wise unsupervisedlearningalgorithm for Deep Belief Networks (DBN), a generative model with many layers of hidden causalvariables. Inthecontextoftheaboveoptimizationproblem,westudythisalgorithmempiricallyandexplorevariantstobetterunderstanditssuccessandextend it to cases where the inputsare continuousor where the structureof the inputdistributionisnotrevealingenoughaboutthevariabletobe predictedina supervised task. Ourexperimentsalsoconfirmthehypothesisthatthegreedylayer-wiseunsupervisedtrainingstrategymostlyhelpstheoptimization,byinitializingweightsina regionnearagoodlocalminimum,givingrisetointernaldistributedrepresentations thatarehigh-levelabstractionsoftheinput,bringingbettergeneralization.