Date of Publication

2-2022

Document Type

Master's Thesis

Degree Name

Master of Science in Computer Science

Subject Categories

Computer Sciences

College

College of Computer Studies

Department/Unit

Computer Science

Honor/Award

Gold Medal for Outstanding Thesis

Thesis Advisor

Arnulfo P. Azcarraga

Defense Panel Chair

Macario O. Cordel, II

Defense Panel Member

Arnulfo P. Azcarraga
Neil Patrick A. Del Gallego

Abstract/Summary

Neural networks are generally considered as function approximation models that map a set of input features to their target outputs. Their approximation capability can be improved through “ensemble learning”. An ensemble of neural networks decreases the error correlation of the group by having each network in the ensemble compensate for the performance of one another. One ensembling technique is the Mixture-of-Experts model, which consists of a set of independently-trained expert neural networks that specialize on their own subset of the dataset, and a gating network that manages the specialization of the expert neural networks. In this model, all the neural networks are trained concurrently, but the expert neural networks are only trained on cases in which they perform well. Some major components of the proposed architecture for this thesis are the Cooperative Ensemble, which trains its neural networks concurrently instead of independently, and the k-Winners-Take-All activation function to drive the specialization among neural network experts on a subset of the input features. This way, there is no longer a need for a centralized gating network to manage the specialization of the neural network experts. We further improve upon the k-Winners-Take-All ensemble neural network by training another neural network with the designated task of learning useful feature representations for the neural networks in the ensemble. To learn such representations, the neural network uses the Soft Nearest Neighbor Loss which engenders a simpler function approximation task for the neural networks in the ensemble. We call the resulting full architecture “Self-Organizing Cooperative Neural Network Experts” (SOCONNE), in which a set of neural networks gain the right to specialize on their own subsets of the dataset without the use of a centralized gating neural network. Numerous experiments on a variety of test datasets show that the novel architecture (1) takes advantage of the learned representations for the set of input features by learning their underlying structure, and (2) uses these learned representations to simplify the task of the neural networks in a cooperative ensemble set-up.

Abstract Format

html

Language

English

Format

Electronic

Physical Description

109 leaves

Keywords

Neural networks (Computer science)

Upload Full Text

wf_yes

Embargo Period

2-26-2022

Download

Share

COinS