Dynamic net
One paradgm is that the network could be dynamic. For such a model we want to be able to add and remove nodes. This could be an issue for networks that are engineered to scale up and down to a certain size. E.g. cnn. If we drop we may like to either rescale pad or introduce a new node. Also dynamic nets should be engineered to handle differnt size of input and still work ok. If a node vanishes drop it.
Also drop any other nodes that are in its product zero. If a node explodes dup it and enforce refularization using drip out at
Also use renormalization to reduce infinities Use an infintessimal rep vanshing nodes We apply a contraction mapping Another idea is that we have a minimumn and maximum weight settings and we renormelize. The weights after propergating the errors.
Game layers
- Should units compete or cooperate.
- Can we make a layer to handle that. If the units cooperate the may tend to overfit. In the sense that they wont lean as effectively as nets whose units compete. This may be done by adjusting the loss. Can we donit using a layer or a reguliser. Ie can we make competition take place at different level s of our net.
This leads to the idea of allowing the net to self orgenize.
Weight sharing. That is a regulizer allowing the net to learn from less data.
Citation
@online{bochman2019,
author = {Bochman, Oren},
title = {Exploding and Vanishing Nodes.},
date = {2019-07-31},
url = {https://orenbochman.github.io/posts/2019/2019-07-31-Exploding-and-vanishing-nodes/2019-07-31-Exploding-and-vanishing-nodes.html},
langid = {en}
}