0

In machine learning regression is usually used to determine the mapping $f(x)$ when you know some values $y$. However I was wondering if some research is already done on determining a functional. Essentially having a function as input, instead of a vector.

Some searching on the web didn't really show me some progress on that subject, but maybe I am not using the right terminology since I am not a ML expert.

Maybe someone has knowledge of this kind of research?

Fork2
  • 223
  • Can you be more specific? Do you have a particular formulation or you are just wondering if there is a sort of generalization? Consider that in general a machine learning problem is substantially a minimization problem of any king. Usually you handle discrete entities but there are no constraints. – Ilis Oct 08 '15 at 17:40
  • Consider a function $\rho$ and a functional $V$ that maps $f\mapsto V[f] \in \mathbb{C}$. No I am wondering how I can obtain this $V$ when I have some knowledge about the output data.

    You can say hat ML is just a minimization problem, but I don't have any knowledge of solving a problem like that.

     – Fork2 Oct 08 '15 at 17:47
  • I cannot help you yet. What are $\rho$ and $f$? Do they live in a Hilbert space or the set of $f$ is endowed with a norm or a distance? – Ilis Oct 08 '15 at 18:00
  • I'm sorry, I should have replaced $f$ by $\rho$. And actually, let's assume it is part of a Hilbert space. – Fork2 Oct 08 '15 at 19:37

1 Answers1

1

In general you know or you chose a family of allowed functional $V$ among which you seek that depends on some parameters, for example the coefficient we try to estimate in the linear regression.

In this scenario, given a map $V=V_\vartheta$ depending on some parameters $\vartheta$, a set of observations $\rho_1,\dots,\rho_n$ and the evaluations $v_i=V_\vartheta (\rho_i)\in \mathbb C$, we want to determine the $\hat\vartheta$ that better fits the training set (our observations). This might be attained in several ways (depending on the particular statement of the problem or the tools you have available,...), but usually the problem is stated as the minimum problem: $$\hat\vartheta =\underset{\vartheta \in \Theta}{\text{argmin}} \left\lVert\sum_{i=1}^n V_\vartheta(\rho_i) - v_i \right\rVert$$ where the norm is to be chosen according to the purpose (for example the $\ell_2$-norm) and $\Theta$ is the set of feasible $\vartheta$ for the problem (for example $\mathbb R^m$).

Additional Example

Consider the problem of finding the best numerical quadrature of a function on an interval $\Omega$. Then you are aiming to obtain the functional $I=\int_\Omega$ that is the an functional $I:L^1(\Omega)\rightarrow \mathbb R$.

Here we can chose to find the best approximation of $I$ over a particular family of functionals dependent on some parameters in which our search becomes more feasible. The family is actual the model that we are confident that fits the setup. Some models for machine learning are linear regression, logistic regression, softmax, CART, ...

For this particular problem we can chose the family of functional that evaluates the function (we want to integrate) in two points $x_1$, $x_0$ (the parameters of the family) and estimating the integral $$I\approx V_{(x_1,x_2)}(f)=(f(x_1)+f(x_2))\frac{|\Omega|}{2}.$$

This is merely a possible adaption of a problem that fits your requirements to the framework I proposed and it might help you to erase your doubts.

Ilis
  • 150
  • Thank you, I understand what you are saying. So the problem is reduced to find a fitted functional $V$ depending on some parameters $\theta$, which is quite a task. – Fork2 Oct 08 '15 at 20:08
  • Exactly. Often it reduces to a least squares problem that is quite representative and it may help you understand the framework. – Ilis Oct 08 '15 at 20:18
  • Let me give you an example. Assume that you have a $\delta$ distribution and use it as a functional. What kind of $V$ would you use? (Ofcourse assuming you don't know that it is a $\delta$ distribution. – Fork2 Oct 08 '15 at 20:32
  • This is actually unfeasible... I dont think you can... I added an example that might help you. – Ilis Oct 08 '15 at 21:16