Anomalies 2 - nonparametric vs. parametric
Also, functions in our heads are probably nonparametric. That is, we do not learn a set of parameters and then use it every time we encounter something. It doesn’t seem right to me. However, think about nonparametric functions. It’s like we learn some rules, heuristics and use it (and/or combine it with other functions) every time we encounter something. That makes more sense. For example, you might find this dumb but we can do classification with nearest neighbors up to certain accuracy for every data set :D. Again, maybe you do not find this interesting but think about it. We do no training at all but come up with this certain rule, certain knowledge that examples that are nearby should be similar. Instead of learning anything, why not store a sufficient amount of experiences and interpolate from them?
I am not advocating that we should just use nonparametric functions but we should appreciate them. It explains a lot. It also explains anomalies to some extent. We can easily change operation mode without constantly checking whether we are in the operation regime or not.