Improving Saliency Metrics for Channel Pruning of Convolutional Neural Networks
Citation:
Persand, Kaveena Devi, Improving Saliency Metrics for Channel Pruning of Convolutional Neural Networks, Trinity College Dublin.School of Computer Science & Statistics, 2022Abstract:
Channel pruning is an effective technique to reduce the size of Convolutional Neural Networks (CNNs). A decisive part of any pruning algorithm is its saliency metric. We propose different techniques to improve saliency metrics for channel pruning.
Saliency metrics are encompassed in a larger pruning algorithm and are expressed in various forms. Without a standard form, it can be difficult to identify
and compare these metrics. To facilitate the comparison of saliency metrics, we
propose a taxonomy based on four independent components: base input, pointwise
metric, reduction, and scaling. We classify existing saliency metrics according to our
proposed taxonomy. We find that new channel saliency metrics can be created using
the components of existing saliency metrics. We also propose a new scaling method.
We evaluate the newly created saliency metrics (using existing components as well
as our new scaling method) and find that some of the metrics outperform existing
ones. We also provide some guidance for the construction of new saliency metrics.
Specifically, we highlight the importance of the reduction and scaling methods.
Pruning algorithms generally rely on a single saliency metric for pruning. Even
if that chosen metric performs well on average, it can make poor decisions from time
to time. Through the use of multiple saliency metrics, we can compensate the poor
decisions of the single metric. We show that the combination of saliency metrics
is possible and combine the decisions of multiple saliency metrics using a myopic
oracle. We show that the decisions of the myopic oracle can lead to better pruning
rates than the constituent metrics.
When pruning one channel from CNNs with split and join connections, more
pruning opportunities become apparent. Multiple channels can be pruned by transitively removing channel weights from other layers of the network. However,
most saliency metrics do not factor in these extra structural constraints. We propose domino saliency metrics, built on top of existing channel saliency metrics, to
factor in these constraints. We show that the use of domino saliency metrics can
significantly improve pruning rates for networks with splits and joins.
Sponsor
Grant Number
Science Foundation Ireland (SFI)
Arm Research
Author's Homepage:
https://tcdlocalportal.tcd.ie/pls/EnterApex/f?p=800:71:0::::P71_USERNAME:PERSANDKDescription:
APPROVED
Author: Persand, Kaveena Devi
Advisor:
Gregg, DavidPublisher:
Trinity College Dublin. School of Computer Science & Statistics. Discipline of Computer ScienceType of material:
ThesisCollections
Availability:
Full text availableKeywords:
Machine Learning, Convolutional Neural Network, PruningMetadata
Show full item recordLicences: