Mathematics Stack Exchange
Mathematics Stack Exchange

Questions tagged [numerical-optimization]

Numerical methods for continuous optimization.

Numerical Optimization is one of the central techniques in Machine Learning. For many problems it is hard to figure out the best solution directly, but it is relatively easy to set up a loss function that measures how good a solution is - and then minimize the parameters of that function to find the solution.

Learn more about solving numerical optimization problems in this pdf.

Sources: http://www.benfrederickson.com/numerical-optimization/

1526 questions
11
votes
4 answers

Image for a strict local minimizer that is not an isolated minimizer

From Numerical Optimization by Nocedal and Wright: A point $x^*$ is a strict local minimizer if there is a neighborhood $N$ of $x^*$ such that $f(x^*) x^*$. A point $x^*$ is an isolated local minimizer if…
O. Altun
  • 365
7
votes
2 answers

Quasi-newton methods: SR1 and BFGS inverse update

In Numerical Optimization by Nocedal and Wright, (http://home.agh.edu.pl/~pba/pdfdoc/Numerical_Optimization.pdf) Chapter 2 on unconstrained optimization, page 25 top, the authors claim that "The equivalent formula for SR1, $$B_{k+1} = B_k +…
user2193268
  • 303
  • 4
  • 11
4
votes
0 answers

Trust region algorithm mystery

I was watching the trust region algorithm implementation in the dlib code, optimization_trust_region.h, where it is stated that "This is an implementation of algorithm 4.3 (Trust Region Subproblem) from the book Numerical Optimization by Nocedal…
Dan
  • 71
2
votes
1 answer

Maximum product of sums of a given array of numbers

Given real numbers $a_1,...,a_n \geq 0$, partition them into groups, so that the total product of each group's sum is maximal: $$M = (a_{i_{1,1}} + ... + a_{i_{k_1,1}}) \cdot ... \cdot (a_{i_{1,l}} + ... + a_{i_{k_l,l}}$$ You may put the numbers in…
Orbit
  • 51
2
votes
1 answer

Applying a quasi newton (L-BFGS) method to a non differentiable cost function.

I'm reading through a paper which presents at some point an optimization step to a function of the form: $$ E = \sum_i \left|\alpha_i - \beta_i \right| $$ where $\alpha_i$ and $\beta_i$ are also functions, it doesn't really matter the specific form.…
user8469759
  • 5,285
2
votes
0 answers

need difficult 2-3dim objective functions to optimize, by algorithm

for teaching purposes, I am looking for continuous compact functions defined over one or two variables that are deliberately chosen to illustrate how optimization algorithms can run into difficulties, i.e., requiring unusually many iterations to…
ivo Welch
  • 121
2
votes
1 answer

why not handle box-constraints with a transformation

I have a question that I've always wondered about concerning the "L-BFGS-B" algorithm. I am not familar with the details of the algorithm except for the fact that it optimizes a non-linear function subject to box-constraints. My question is: Given…
mark leeds
  • 1,514
1
vote
1 answer

bad convergence of steepest descent

Under which circumstances does the steepest descent method converge badly? I know if the search direction is approximately perpendicular to the descent direction the steepest descent method converges slowly. What other situations are there in which…
dinosaur
  • 2,252
1
vote
0 answers

Convex Optimization Proof

I am a newbie to SE. I am a computer science undergrad pursuing my masters in Data Science. I am trying to prove the following result. For $i=1,\ldots,m$, let $\alpha_i\in\mathbb{R}^n$ and $c_i>0$. Put $f=\sum c_c \exp((\alpha_i,x))$. Show that…
Nasreen Ahmed
  • 65
1
vote
1 answer

Maximize a function with a condition.

The problem is placing $n$ number of points on a sphere in $\mathbb{R}^m$, as far apart as possible from eachother. I want to formulate this problem as an optimizationproblem where the objectfunction is the function that describes the sum of…
Parseval
  • 6,413
1
vote
2 answers

Nesterov's momentum derivation

On page 75 in Sutskever's thesis http://www.cs.utoronto.ca/~ilya/pubs/ilya_sutskever_phd_thesis.pdf In equation (7.5) setting $a_0=1$, $a_{t+1} = (1+\sqrt{4 a_t^2 + 1})/2 $ The author said, "to understand the sequence $a_t$ we note that the function…
Patrick
  • 356
1
vote
1 answer

For an angle, what does it mean to be bounded away 90 degrees?

Nocedal and Wright, in their book Numerical Optimization, just above (3.17), use this wording: ...the angle $\theta_k$ is bounded away from $90^\circ$,... what does this mean?
O. Altun
  • 365
1
vote
0 answers

Powell vs Levmar

I'm learning about the application of numerical optimization, and noticed that in one of the programs that I'm using it uses Powell's method to get parameters of a piece-wise function, and Levmar if the function can be expressed as a simple formula…
dev_nut
  • 161
  • 3
  • 11
1
vote
0 answers

Levenberg-Marquardt - What is preferable (A + mu.I) or (A + mu.diag[A])?

The step size is computed by solving (A + mu I) h = -g I could find in some literature that one can compute the step size by solving (A + mu A') h = -g where, A' = diagonal(A) It is said that this is helpful for error valley problems, where the…
Umesh Tiwari
  • 11
0
votes
0 answers

Brachistochrone involving gravitational changes dependent on x

as an extension to the normal brachistochrone problem: $$T[y]=F(y,y')= \tfrac{1}{\sqrt{2g}}\int_{0}^{x_{b}}\tfrac{\sqrt{1+(y'(x))^2}}{\sqrt{y(x)}}dx$$ I was asked to get the gravity dependent on x, so the new formula would be $$T[y]=F(y,y')=…
Bastian Sommerfeld
  • 33
  • 4
1
2 3