1

In the natural gradient paper, the first line writes:

The parameter space of neural networks has a Riemannian metric structure.

I don't know why. I has basic knowledge of Riemannian metric: it defines the metric of local space of riemanian space. And I know riemannian space is a curved space. But I don't get it that how is the NN parameter space a curved space rather the Euclidean space?

I have read the correspoonding material mentioned in the paper briefly, it seems to regard NN as a probability density functions which is in L1 space, suggesting that the parameter space is constrained to make the function be in L1. I am fine with the explaination. But as I know, the parameters of a network is not necessarily under some constraint. And I doubt NN is a L1 function because usually we just apply softmax layer to the raw output of the NN to make the output within (0,1).

Paul Frost
  • 76,394
  • 12
  • 43
  • 125
zhixin
  • 59
  • 4
  • The first paragraph of the paper mentions that this is introduced in Amari (1985). Have you checked there? – Arkady Oct 08 '23 at 04:23
  • @Arkady Thanks for the hint. I have read the material briefly, it seems to regard NN as a probability density functions which is in L1 space, suggesting that the parameter space is constrained to make the function be in L1. I am fine with the explaination. But as i know, the parameters of a network is not necessarily under some constraint. And I doubt NN is a L1 function because usually we just apply softmax layer to the raw output of the NN. – zhixin Oct 08 '23 at 07:27

0 Answers0