One of the most basic and essential results in differential calculus known as the “first derivative theorem” states that the derivative of a function at an extremum in an open interval inside the domain of the function vanishes. Note that in the post on the extreme value theorem, we showed that a continuous real-valued function…

# Category: Mathematical Optimization

## Tangent line to a curve

Every standard calculus textbook discusses the geometric interpretation of the tangent line to a curve at a given point. However, we define the tangent line differently. The tangent line to a curve at a given point is the line which provides the best local approximation of all the lines passing through the given point. In…

## Regression line

A regression line is a line that fits best to a set of numerical data points in the plane. In this post, we will discuss the concept of a regression (least squares) line and a method for finding it. In statistics, regression is an analytical method to measure the association of one or more independent…

## Least squares problem

The least squares problem is to obtain a vector that the distance of the value of the given linear map at that vector from another given vector is minimized. More specifically, let \(A\) be an \(m \times n\) real matrix and \(b\) a vector in \(\mathbb{R}^m\). The least squares problem is to find a suitable…

## Extreme value theorem

The extreme value theorem states that a continuous real-valued function defined over a closed interval attains its minimum and maximum at some points in the closed interval. In other words, if \(f\) is a real-valued function continuous on a closed interval \([a,b]\), then by the extreme value theorem, there are two points \(l\) and \(u\)…

## Vector orthogonal projection

Vector orthogonal projection is an essential operation in the Gram-Schmidt orthonormalizing process and the least square problem. In this post, we discuss vector orthogonal projection, give some suitable examples, and solve some exercises. A motivational example of a vector orthogonal projection in analytic geometry Consider the point \(P=(1,7)\) and the line \(L\) passing through the…

## Matrix norms and examples

The main purpose of this post is to introduce different matrix norms such as Frobenius norm and \(p\)-norms. In the following, first we give a general definition of matrix norms. General definition of matrix norms In the post on matrix operations, it is explained that \(M_{n\times n}(\mathbb{R})\) is a real vector space. A matrix norm…

## Norms of vector spaces

Sometimes on a single vector space, different norms can be defined. In this post, first we give the definition of an inner product vector space. Then, we proceed to discuss different norms on the finite-dimensional real vector spaces, i.e. the vector spaces of the form \(\mathbb{R}^n\). The dot product In the post on the dot…

## Dot product of vectors

Definition of the dot product in the two-dimensional space The definition of the dot product of two vectors \(u = (u_1,u_2)\) and \(v = (v_1,v_2)\), denoted by \( u \cdot v \), is as follows: $$ u \cdot v = u_1 v_1 + u_2 v_2. $$ Example. Let us compute the dot product of \(u…