Simple but effective GRU variants
Abstract
Recurrent Neural Network (RNN) is a widely used deep learning architecture applied to sequence learning problems. However, it is recognized that RNNs suffer from exploding and vanishing gradient problems that prohibit the early layers of the network from learning the gradient information. GRU networks are particular kinds of recurrent networks that reduce the short-comings of these problems. In this study, we propose two variants of the standard GRU with simple but effective modifications. We applied an empirical approach and tried to determine the effectiveness of the current units and recurrent units of gates by giving different coefficients. Interestingly, we realize that applying such minor and simple changes to the standard GRU provides notable improvements. We comparatively evaluate the standard GRU with the proposed two variants on four different tasks: (1) sentiment classification on the IMDB movie review dataset, (2) language modeling task on Penn TreeBank (PTB) dataset, (3) sequence to sequence addition problem, and (4) question answering problem on Facebook's bAbitasks dataset. The evaluation results indicate that the proposed two variants of GRU consistently outperform standard GRU. © 2021 IEEE.
Collections
Related items
Showing items related by title, author, creator and subject.
-
Recurrent cholangitis associated with biliary sludge and Phrygian cap anomaly diagnosed by magnetic resonance imaging and magnetic resonance cholangiopancreatography despite normal ultrasound and computed tomography
Başaranoğlu, Metin; Balcı, Numan Cem (Taylor & Francis, 2005)A 31-year-old woman presented with a one and half years' history of intermittent right upper quadrant (RUQ) pain high fever and severely painful warm and reddish swollen skin lesions on the fingers. Acute attack resolution ... -
Repair of recurrent patent ductus arteriosus in an adult with cardiopulmonary bypass
Arbatlı, Harun; Özbek, Uğur; Demirsoy, Ergun; Ünal, Mehmet; Yağan, Naci; Sönmez, Bingür (Blackwell Futura Publishing Inc, 2003)Recurrence of ductal patency is a rarely encountered complication in surgical repair of patent ductus arteriosus (PDA). An adult patient with ductal recurrency underwent closure of ductus by using cardiopulmonary bypass ... -
Mobile Application Development for the Estimation of Recurrence in Post-Operative Kidney Cancer Cases
Tander, Baran; Özmen, Atilla; Ozden, Ender (IEEE, 2018)In this paper a post-operative recurrence estimation tool called Sorbellini's nomogram for the kidney cancer patients showing no metastates is introduced and a novel application for mobile devices based on this model is ...