You can decompose your function as
$$
f = \kappa \circ h \circ g
$$
where
$$
g(X) =X^{-1} \quad ; \quad h(X) = Xv .
$$
In differential form
$$
d \kappa = \langle \nabla \kappa(\mathbb{x}), d\mathbb{x} \rangle
\quad ; \quad
d h = (dX) v \quad ; \quad
d g = - X^{-1} (dX) X^{-1} .
$$
Then, by applying chain rule we get differental of $f$
$$
d f = -\Big\langle \nabla \kappa(A^{-1}v), A^{-1} (dA) A^{-1} v \Big\rangle.
$$
You can compute derivative in form of matrix
$$
\frac{\partial f}{\partial A} (A) = (x_{i,j} )^n_{i,j = 1},
$$
where each entry has a value
$$
x_{i,j} = -\Big\langle \nabla \kappa(A^{-1}v), A^{-1} X_{i,j} A^{-1} v \Big \rangle
$$
with $X_{i,j}$ being a matrix with $1$ at position $i,j$ and $0$ everywhere else.