Файл:Recurent neural network regularization 1409.2329v5.pdf

Материал из Материалы по машинному обучению
Перейти к: навигация, поиск
Recurent_neural_network_regularization_1409.2329v5.pdf(0 × 0 пикселей, размер файла: 115 КБ, MIME-тип: application/pdf)
  • Wojciech Zaremba∗ New York University woj.zaremba@gmail.com
  • Ilya Sutskever, Oriol Vinyals Google Brain {ilyasu,vinyals}@google.com

ABSTRACT

We present a simple regularization technique for Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units. Dropout, the most successful technique for regularizing neural networks, does not work well with RNNs and LSTMs. In this paper, we show how to correctly apply dropout to LSTMs, and show that it substantially reduces overfitting on a variety of tasks. These tasks include language modeling, speech recognition, image caption generation, and machine translation.

INTRODUCTION

The Recurrent Neural Network (RNN) is neural sequence model that achieves state of the art performance on important tasks that include language modeling Mikolov (2012), speech recognition Graves et al. (2013), and machine translation Kalchbrenner & Blunsom (2013). It is known that successful applications of neural networks require good regularization. Unfortunately, dropout Srivastava (2013), the most powerful regularization method for feedforward neural networks, does not work well with RNNs. As a result, practical applications of RNNs often use models that are too small because large RNNs tend to overfit. Existing regularization methods give relatively small improvements for RNNs Graves (2013). In this work, we show that dropout, when correctly used, greatly reduces overfitting in LSTMs, and evaluate it on three different problems.


The code for this work can be found in https://github.com/wojzaremba/lstm

История файла

Нажмите на дату/время, чтобы просмотреть, как тогда выглядел файл.

Дата/времяРазмерыУчастникПримечание
текущий13:42, 21 декабря 20160 × 0 (115 КБ)Slikos (обсуждение | вклад)RECURRENT NEURAL NETWORK REGULARIZATION
  • Вы не можете перезаписать этот файл.

Следующие 2 страницы ссылаются на данный файл: