I guess the answer is: $C(n,n-3)/2$ since for $n$-sided polygon,there are $n$ vertices, and for each vertex, it cannot form diagonal with the adjacent points and itself. So, each vertex can form $n-3$ points. And half of the vertices have repeated diagonals and we need to divide by $2$.
It holds true for quadrilateral and pentagon,but why it does not fit triangles?