least-squares method

• 最小二乗法

関

least-squares analysis

WordNet

1. raise to the second power
2. (geometry) a plane rectangle with four equal sides and four right angles; a four-sided regular polygon; "you can compute the area of a square if you know the length of its sides" (同)foursquare
3. the product of two equal terms; "nine is the second power of three"; "gravity is inversely proportional to the square of the distance" (同)second power
4. make square; "Square the circle"; "square the wood with a file" (同)square_up
5. something approximating the shape of a square
6. a hand tool consisting of two straight arms at right angles; used to construct or test right angles; "the carpenter who built this room must have lost his square"
7. any artifact having a shape similar to a plane geometric figure with four equal sides and four right angles; "a checkerboard has 64 squares"
8. someone who doesnt understand what is going on (同)lame
9. a formal and conservative person with old-fashioned views (同)square toes
10. rigidly conventional or old-fashioned (同)straight
11. without evasion or compromise; "a square contradiction"; "he is not being as straightforward as it appears" (同)straightforward, straight
12. be compatible with; "one idea squares with another"
13. cause to match, as of ideas or acts
14. having four equal sides and four right angles or forming a right angle; "a square peg in a round hole"; "a square corner"
15. leaving no balance; "my account with you is now all square"
16. pay someone and settle a debt; "I squared with him"
17. position so as to be square; "He squared his shoulders"
18. the superlative of `little that can be used with mass nouns and is usually preceded by `the; a quantifier meaning smallest in amount or extent or degree; "didnt care the least bit"; "he has the least talent of anyone"
19. something that is of no importance; "it is the least I can do"; "that is the least of my concerns"
20. used to form the superlative; "The garter snake is the least dangerous snake" (同)to the lowest degree
21. a way of doing something, especially a systematic way; implies an orderly logical arrangement (usually in steps)
22. a unit of length of thread or yarn

PrepTutorEJDIC

1. 『正方形』;四角な物;(チェス・チェッカーなどの盤の,正方形の)目,ます目 / (四角い)『広場』(街路の交差点にあって,しばしば中央に植木や芝などが植えてあり,小公園になっている);《おもに英》広場の回りの建物(街路)(《略》Sq.) / (四方を街路で囲まれた方形の)一区画,ブロック / 直角定規,かね尺 / (数の)『2乗』,平方(《略》sq.) / 《俗》旧式な人 / 『正方形の』,四角な,直角の,直角をなす / 角ばった,がっかりした / 『平方の』,2乗の(《略》『sq.』) / 《補語にのみ用いて》対等の,五分五分の(even);貸し借りにない / 正直な(honest),公正な(fair),正しい(just) / 率直な,はっきりした,きっぱりした(direct) / 実質のある,十分な / 《俗》しゃちほこばった / =squarely / …‘を'正方形(四角)にする;直角にする / …‘を'正方形(四角)に区切る《+『off』+『名』,+『名』+『off』》 / 〈肩・ひじなど〉‘を'張る / (人と)〈勘定〉‘を'決済する,清算する《+『名』〈勘定〉+『with』+『名』〈人〉》 / (…と)…‘を'一致させる,適合させる《+『名』+『with』+『名』》 / 《受動態で》〈数〉‘を'2乗する;〈ある形・図形など〉‘の'平方積(面積)を求める / 〈人〉‘を'買収する,抱き込む,…‘に'わいろを使う / 〈試合の得点〉‘を'同点にする / (…と)一致する,適合する《+『with』+『名』》
2. (littleの最上級;比較級はless) / (大きさ・程度が)『最も小さい』(『少ない』) / (重要性・身分が)最も低い / (大きさ・程度・重要性が)『最小』(『最少』,『最小限』)『の物事』 / 『最も少なく』(『低く』)
3. 〈C〉(特に秩序だった)(…の)『方法』,方式《+『of』+『名』(do『ing』)》 / 〈C〉〈U〉(思考行・行為・行動の)きちょうめんさ;秩序,筋道
4. 草原,牧草地(meadow)

Wikipedia preview

出典(authority):フリー百科事典『ウィキペディア（Wikipedia）』「2017/10/17 10:04:31」(JST)

wiki en

[Wiki en表示]

Part of a series on Statistics
Regression analysis
Models
Linear regression Simple regression Ordinary least squares Polynomial regression General linear model
Generalized linear model Discrete choice Logistic regression Multinomial logit Mixed logit Probit Multinomial probit Ordered logit Ordered probit Poisson
Multilevel model Fixed effects Random effects Mixed model
Nonlinear regression Nonparametric Semiparametric Robust Quantile Isotonic Principal components Least angle Local Segmented
Errors-in-variables
Estimation
Least squares Ordinary least squares Linear (math) Partial Total Generalized Weighted Non-linear Non-negative Iteratively reweighted Ridge regression
Least absolute deviations Bayesian Bayesian multivariate
Background
Regression model validation Mean and predicted response Errors and residuals Goodness of fit Studentized residual Gauss–Markov theorem
Statistics portal

The result of fitting a set of data points with a quadratic function

Conic fitting a set of points using least-squares approximation

The method of least squares is a standard approach in regression analysis to the approximate solution of overdetermined systems, i.e., sets of equations in which there are more equations than unknowns. "Least squares" means that the overall solution minimizes the sum of the squares of the residuals made in the results of every single equation.

The most important application is in data fitting. The best fit in the least-squares sense minimizes the sum of squared residuals (a residual being: the difference between an observed value, and the fitted value provided by a model). When the problem has substantial uncertainties in the independent variable (the x variable), then simple regression and least-squares methods have problems; in such cases, the methodology required for fitting errors-in-variables models may be considered instead of that for least squares.

Least-squares problems fall into two categories: linear or ordinary least squares and nonlinear least squares, depending on whether or not the residuals are linear in all unknowns. The linear least-squares problem occurs in statistical regression analysis; it has a closed-form solution. The nonlinear problem is usually solved by iterative refinement; at each iteration the system is approximated by a linear one, and thus the core calculation is similar in both cases.

Polynomial least squares describes the variance in a prediction of the dependent variable as a function of the independent variable and the deviations from the fitted curve.

When the observations come from an exponential family and mild conditions are satisfied, least-squares estimates and maximum-likelihood estimates are identical. The method of least squares can also be derived as a method of moments estimator.

The following discussion is mostly presented in terms of linear functions but the use of least squares is valid and practical for more general families of functions. Also, by iteratively applying local quadratic approximation to the likelihood (through the Fisher information), the least-squares method may be used to fit a generalized linear model.

For the topic of approximating a function by a sum of others using an objective function based on squared distances, see least squares (function approximation).

The least-squares method is usually credited to Carl Friedrich Gauss (1795), but it was first published by Adrien-Marie Legendre.

History

Context

The method of least squares grew out of the fields of astronomy and geodesy, as scientists and mathematicians sought to provide solutions to the challenges of navigating the Earth's oceans during the Age of Exploration. The accurate description of the behavior of celestial bodies was the key to enabling ships to sail in open seas, where sailors could no longer rely on land sightings for navigation.

The method was the culmination of several advances that took place during the course of the eighteenth century:

• The combination of different observations as being the best estimate of the true value; errors decrease with aggregation rather than increase, perhaps first expressed by Roger Cotes in 1722.
• The combination of different observations taken under the same conditions contrary to simply trying one's best to observe and record a single observation accurately. The approach was known as the method of averages. This approach was notably used by Tobias Mayer while studying the librations of the moon in 1750, and by Pierre-Simon Laplace in his work in explaining the differences in motion of Jupiter and Saturn in 1788.
• The combination of different observations taken under different conditions. The method came to be known as the method of least absolute deviation. It was notably performed by Roger Joseph Boscovich in his work on the shape of the earth in 1757 and by Pierre-Simon Laplace for the same problem in 1799.
• The development of a criterion that can be evaluated to determine when the solution with the minimum error has been achieved. Laplace tried to specify a mathematical form of the probability density for the errors and define a method of estimation that minimizes the error of estimation. For this purpose, Laplace used a symmetric two-sided exponential distribution we now call Laplace distribution to model the error distribution, and used the sum of absolute deviation as error of estimation. He felt these to be the simplest assumptions he could make, and he had hoped to obtain the arithmetic mean as the best estimate. Instead, his estimator was the posterior median.

The method

Carl Friedrich Gauss

The first clear and concise exposition of the method of least squares was published by Legendre in 1805. The technique is described as an algebraic procedure for fitting linear equations to data and Legendre demonstrates the new method by analyzing the same data as Laplace for the shape of the earth. The value of Legendre's method of least squares was immediately recognized by leading astronomers and geodesists of the time.

In 1809 Carl Friedrich Gauss published his method of calculating the orbits of celestial bodies. In that work he claimed to have been in possession of the method of least squares since 1795. This naturally led to a priority dispute with Legendre. However, to Gauss's credit, he went beyond Legendre and succeeded in connecting the method of least squares with the principles of probability and to the normal distribution. He had managed to complete Laplace's program of specifying a mathematical form of the probability density for the observations, depending on a finite number of unknown parameters, and define a method of estimation that minimizes the error of estimation. Gauss showed that arithmetic mean is indeed the best estimate of the location parameter by changing both the probability density and the method of estimation. He then turned the problem around by asking what form the density should have and what method of estimation should be used to get the arithmetic mean as estimate of the location parameter. In this attempt, he invented the normal distribution.

An early demonstration of the strength of Gauss' method came when it was used to predict the future location of the newly discovered asteroid Ceres. On 1 January 1801, the Italian astronomer Giuseppe Piazzi discovered Ceres and was able to track its path for 40 days before it was lost in the glare of the sun. Based on these data, astronomers desired to determine the location of Ceres after it emerged from behind the sun without solving Kepler's complicated nonlinear equations of planetary motion. The only predictions that successfully allowed Hungarian astronomer Franz Xaver von Zach to relocate Ceres were those performed by the 24-year-old Gauss using least-squares analysis.

In 1810, after reading Gauss's work, Laplace, after proving the central limit theorem, used it to give a large sample justification for the method of least square and the normal distribution. In 1822, Gauss was able to state that the least-squares approach to regression analysis is optimal in the sense that in a linear model where the errors have a mean of zero, are uncorrelated, and have equal variances, the best linear unbiased estimator of the coefficients is the least-squares estimator. This result is known as the Gauss–Markov theorem.

The idea of least-squares analysis was also independently formulated by the American Robert Adrain in 1808. In the next two centuries workers in the theory of errors and in statistics found many different ways of implementing least squares.

Problem statement

The objective consists of adjusting the parameters of a model function to best fit a data set. A simple data set consists of n points (data pairs) $(x_{i},y_{i})\!$, i = 1, ..., n, where $x_{i}\!$ is an independent variable and $y_{i}\!$ is a dependent variable whose value is found by observation. The model function has the form $f(x,\beta )$, where m adjustable parameters are held in the vector ${\boldsymbol {\beta }}$. The goal is to find the parameter values for the model that "best" fits the data. The least-squares method finds its optimum when the sum, S, of squared residuals

$S=\sum _{i=1}^{n}{r_{i}}^{2}$

is a minimum. A residual is defined as the difference between the actual value of the dependent variable and the value predicted by the model. Each data point has one residual. Both the sum and the mean of the residuals are equal to zero.

$r_{i}=y_{i}-f(x_{i},{\boldsymbol {\beta }}).$

An example of a model is that of the straight line in two dimensions. Denoting the y-intercept as $\beta _{0}$ and the slope as $\beta _{1}$, the model function is given by $f(x,{\boldsymbol {\beta }})=\beta _{0}+\beta _{1}x$. See linear least squares for a fully worked out example of this model.

A data point may consist of more than one independent variable. For example, when fitting a plane to a set of height measurements, the plane is a function of two independent variables, x and z, say. In the most general case there may be one or more independent variables and one or more dependent variables at each data point.

Limitations

This regression formulation considers only residuals in the dependent variable. There are two rather different contexts in which different implications apply:

• Regression for prediction. Here a model is fitted to provide a prediction rule for application in a similar situation to which the data used for fitting apply. Here the dependent variables corresponding to such future application would be subject to the same types of observation error as those in the data used for fitting. It is therefore logically consistent to use the least-squares prediction rule for such data.
• Regression for fitting a "true relationship". In standard regression analysis, that leads to fitting by least squares, there is an implicit assumption that errors in the independent variable are zero or strictly controlled so as to be negligible. When errors in the independent variable are non-negligible, models of measurement error can be used; such methods can lead to parameter estimates, hypothesis testing and confidence intervals that take into account the presence of observation errors in the independent variables. An alternative approach is to fit a model by total least squares; this can be viewed as taking a pragmatic approach to balancing the effects of the different sources of error in formulating an objective function for use in model-fitting.

Solving the least squares problem

The minimum of the sum of squares is found by setting the gradient to zero. Since the model contains m parameters, there are m gradient equations:

${\frac {\partial S}{\partial \beta _{j}}}=2\sum _{i}r_{i}{\frac {\partial r_{i}}{\partial \beta _{j}}}=0,\ j=1,\ldots ,m,$

and since $r_{i}=y_{i}-f(x_{i},{\boldsymbol {\beta }})$, the gradient equations become

$-2\sum _{i}r_{i}{\frac {\partial f(x_{i},{\boldsymbol {\beta }})}{\partial \beta _{j}}}=0,\ j=1,\ldots ,m.$

The gradient equations apply to all least squares problems. Each particular problem requires particular expressions for the model and its partial derivatives.

Linear least squares

Main article: Linear least squares

A regression model is a linear one when the model comprises a linear combination of the parameters, i.e.,

$f(x,\beta )=\sum _{j=1}^{m}\beta _{j}\phi _{j}(x),$

where the function $\phi _{j}$ is a function of $x$.

Letting

$X_{ij}={\frac {\partial f(x_{i},{\boldsymbol {\beta }})}{\partial \beta _{j}}}=\phi _{j}(x_{i}),$

we can then see that in that case the least square estimate (or estimator, in the context of a random sample), ${\boldsymbol {\beta }}$ is given by

${\boldsymbol {\hat {\beta }}}=(X^{T}X)^{-1}X^{T}{\boldsymbol {y}}.$

For a derivation of this estimate see Linear least squares (mathematics).

Non-linear least squares

Main article: Non-linear least squares

There is, in some cases, a closed-form solution to a non-linear least squares problem – but in general there is not. In the case of no closed-form solution, numerical algorithms are used to find the value of the parameters $\beta$ that minimizes the objective. Most algorithms involve choosing initial values for the parameters. Then, the parameters are refined iteratively, that is, the values are obtained by successive approximation:

${\beta _{j}}^{k+1}={\beta _{j}}^{k}+\Delta \beta _{j},$

where a superscript k is an iteration number, and the vector of increments $\Delta \beta _{j}$ is called the shift vector. In some commonly used algorithms, at each iteration the model may be linearized by approximation to a first-order Taylor series expansion about ${\boldsymbol {\beta }}^{k}$:

${\begin{aligned}f(x_{i},{\boldsymbol {\beta }})&=f^{k}(x_{i},{\boldsymbol {\beta }})+\sum _{j}{\frac {\partial f(x_{i},{\boldsymbol {\beta }})}{\partial \beta _{j}}}\left(\beta _{j}-{\beta _{j}}^{k}\right)\\&=f^{k}(x_{i},{\boldsymbol {\beta }})+\sum _{j}J_{ij}\,\Delta \beta _{j}.\end{aligned}}$

The Jacobian J is a function of constants, the independent variable and the parameters, so it changes from one iteration to the next. The residuals are given by

$r_{i}=y_{i}-f^{k}(x_{i},{\boldsymbol {\beta }})-\sum _{k=1}^{m}J_{ik}\,\Delta \beta _{k}=\Delta y_{i}-\sum _{j=1}^{m}J_{ij}\,\Delta \beta _{j}.$

To minimize the sum of squares of $r_{i}$, the gradient equation is set to zero and solved for $\Delta \beta _{j}$:

$-2\sum _{i=1}^{n}J_{ij}\left(\Delta y_{i}-\sum _{k=1}^{m}J_{ik}\,\Delta \beta _{k}\right)=0,$

which, on rearrangement, become m simultaneous linear equations, the normal equations:

$\sum _{i=1}^{n}\sum _{k=1}^{m}J_{ij}J_{ik}\,\Delta \beta _{k}=\sum _{i=1}^{n}J_{ij}\,\Delta y_{i}\qquad (j=1,\ldots ,m).$

The normal equations are written in matrix notation as

$\mathbf {(J^{T}J)\,\Delta {\boldsymbol {\beta }}=J^{T}\,\Delta y} .\,$

These are the defining equations of the Gauss–Newton algorithm.

Differences between linear and nonlinear least squares

• The model function, f, in LLSQ (linear least squares) is a linear combination of parameters of the form $f=X_{i1}\beta _{1}+X_{i2}\beta _{2}+\cdots$ The model may represent a straight line, a parabola or any other linear combination of functions. In NLLSQ (nonlinear least squares) the parameters appear as functions, such as $\beta ^{2},e^{\beta x}$ and so forth. If the derivatives $\partial f/\partial \beta _{j}$ are either constant or depend only on the values of the independent variable, the model is linear in the parameters. Otherwise the model is nonlinear.
• Algorithms for finding the solution to a NLLSQ problem require initial values for the parameters, LLSQ does not.
• Like LLSQ, solution algorithms for NLLSQ often require that the Jacobian can be calculated. Analytical expressions for the partial derivatives can be complicated. If analytical expressions are impossible to obtain either the partial derivatives must be calculated by numerical approximation or an estimate must be made of the Jacobian.
• In NLLSQ non-convergence (failure of the algorithm to find a minimum) is a common phenomenon whereas the LLSQ is globally concave so non-convergence is not an issue.
• NLLSQ is usually an iterative process. The iterative process has to be terminated when a convergence criterion is satisfied. LLSQ solutions can be computed using direct methods, although problems with large numbers of parameters are typically solved with iterative methods, such as the Gauss–Seidel method.
• In LLSQ the solution is unique, but in NLLSQ there may be multiple minima in the sum of squares.
• Under the condition that the errors are uncorrelated with the predictor variables, LLSQ yields unbiased estimates, but even under that condition NLLSQ estimates are generally biased.

These differences must be considered whenever the solution to a nonlinear least squares problem is being sought.

Least squares, regression analysis and statistics

The method of least squares is often used to generate estimators and other statistics in regression analysis.

Consider a simple example drawn from physics. A spring should obey Hooke's law which states that the extension of a spring y is proportional to the force, F, applied to it.

$y=f(F,k)=kF\!$

constitutes the model, where F is the independent variable. To estimate the force constant, k, a series of n measurements with different forces will produce a set of data, $(F_{i},y_{i}),\ i=1,\dots ,n\!$, where y_i is a measured spring extension. Each experimental observation will contain some error. If we denote this error $\varepsilon$, we may specify an empirical model for our observations,

$y_{i}=kF_{i}+\varepsilon _{i}.\,$

There are many methods we might use to estimate the unknown parameter k. Noting that the n equations in the m variables in our data comprise an overdetermined system with one unknown and n equations, we may choose to estimate k using least squares. The sum of squares to be minimized is

$S=\sum _{i=1}^{n}(y_{i}-kF_{i})^{2}.$

The least squares estimate of the force constant, k, is given by

${\hat {k}}={\frac {\sum _{i}F_{i}y_{i}}{\sum _{i}F_{i}^{2}}}.$

Here it is assumed that application of the force causes the spring to expand and, having derived the force constant by least squares fitting, the extension can be predicted from Hooke's law.

In regression analysis the researcher specifies an empirical model. For example, a very common model is the straight line model which is used to test if there is a linear relationship between dependent and independent variable. If a linear relationship is found to exist, the variables are said to be correlated. However, correlation does not prove causation, as both variables may be correlated with other, hidden, variables, or the dependent variable may "reverse" cause the independent variables, or the variables may be otherwise spuriously correlated. For example, suppose there is a correlation between deaths by drowning and the volume of ice cream sales at a particular beach. Yet, both the number of people going swimming and the volume of ice cream sales increase as the weather gets hotter, and presumably the number of deaths by drowning is correlated with the number of people going swimming. Perhaps an increase in swimmers causes both the other variables to increase.

In order to make statistical tests on the results it is necessary to make assumptions about the nature of the experimental errors. A common (but not necessary) assumption is that the errors belong to a normal distribution. The central limit theorem supports the idea that this is a good approximation in many cases.

• The Gauss–Markov theorem. In a linear model in which the errors have expectation zero conditional on the independent variables, are uncorrelated and have equal variances, the best linear unbiased estimator of any linear combination of the observations, is its least-squares estimator. "Best" means that the least squares estimators of the parameters have minimum variance. The assumption of equal variance is valid when the errors all belong to the same distribution.
• In a linear model, if the errors belong to a normal distribution the least squares estimators are also the maximum likelihood estimators.

However, if the errors are not normally distributed, a central limit theorem often nonetheless implies that the parameter estimates will be approximately normally distributed so long as the sample is reasonably large. For this reason, given the important property that the error mean is independent of the independent variables, the distribution of the error term is not an important issue in regression analysis. Specifically, it is not typically important whether the error term follows a normal distribution.

In a least squares calculation with unit weights, or in linear regression, the variance on the jth parameter, denoted $\operatorname {var} ({\hat {\beta }}_{j})$, is usually estimated with

$\operatorname {var} ({\hat {\beta }}_{j})=\sigma ^{2}([X^{T}X]^{-1})_{jj}\approx {\frac {S}{n-m}}([X^{T}X]^{-1})_{jj},$

where the true error variance σ² is replaced by an estimate based on the minimised value of the sum of squares objective function S. The denominator, n − m, is the statistical degrees of freedom; see effective degrees of freedom for generalizations.

Confidence limits can be found if the probability distribution of the parameters is known, or an asymptotic approximation is made, or assumed. Likewise statistical tests on the residuals can be made if the probability distribution of the residuals is known or assumed. The probability distribution of any linear combination of the dependent variables can be derived if the probability distribution of experimental errors is known or assumed. Inference is particularly straightforward if the errors are assumed to follow a normal distribution, which implies that the parameter estimates and residuals will also be normally distributed conditional on the values of the independent variables.

Weighted least squares

See also: Weighted mean and Linear least squares (mathematics) § Weighted linear least squares

A special case of generalized least squares called weighted least squares occurs when all the off-diagonal entries of Ω (the correlation matrix of the residuals) are null; the variances of the observations (along the covariance matrix diagonal) may still be unequal (heteroscedasticity).

The expressions given above are based on the implicit assumption that the errors are uncorrelated with each other and with the independent variables and have equal variance. The Gauss–Markov theorem shows that, when this is so, ${\hat {\boldsymbol {\beta }}}$ is a best linear unbiased estimator (BLUE). If, however, the measurements are uncorrelated but have different uncertainties, a modified approach might be adopted. Aitken showed that when a weighted sum of squared residuals is minimized, ${\hat {\boldsymbol {\beta }}}$ is the BLUE if each weight is equal to the reciprocal of the variance of the measurement

$S=\sum _{i=1}^{n}W_{ii}{r_{i}}^{2},\qquad W_{ii}={\frac {1}{{\sigma _{i}}^{2}}}$

The gradient equations for this sum of squares are

$-2\sum _{i}W_{ii}{\frac {\partial f(x_{i},{\boldsymbol {\beta }})}{\partial \beta _{j}}}r_{i}=0,\qquad j=1,\ldots ,n$

which, in a linear least squares system give the modified normal equations,

$\sum _{i=1}^{n}\sum _{k=1}^{m}X_{ij}W_{ii}X_{ik}{\hat {\beta }}_{k}=\sum _{i=1}^{n}X_{ij}W_{ii}y_{i},\qquad j=1,\ldots ,m\,.$

When the observational errors are uncorrelated and the weight matrix, W, is diagonal, these may be written as

$\mathbf {\left(X^{T}WX\right){\hat {\boldsymbol {\beta }}}=X^{T}Wy} .$

If the errors are correlated, the resulting estimator is the BLUE if the weight matrix is equal to the inverse of the variance-covariance matrix of the observations.

When the errors are uncorrelated, it is convenient to simplify the calculations to factor the weight matrix as $w_{ii}={\sqrt {W_{ii}}}$. The normal equations can then be written in the same form as ordinary least squares:

$\mathbf {\left(X'^{T}X'\right){\hat {\boldsymbol {\beta }}}=X'^{T}y'} \,$

where we define the following scaled matrix and vector:

${\begin{aligned}\mathbf {X'} &=\operatorname {diag} \left(\mathbf {w} \right)\mathbf {X} ,\\\mathbf {y'} &=\operatorname {diag} \left(\mathbf {w} \right)\mathbf {y} =\mathbf {y} \oslash \mathbf {\sigma } .\\\end{aligned}}$

This is a type of whitening transformation; the last expression involves an entrywise division.

For non-linear least squares systems a similar argument shows that the normal equations should be modified as follows.

$\mathbf {(J^{T}WJ)\,{\boldsymbol {\Delta }}\beta =J^{T}W\,{\boldsymbol {\Delta }}y} .\,$

Note that for empirical tests, the appropriate W is not known for sure and must be estimated. For this feasible generalized least squares (FGLS) techniques may be used.

Relationship to principal components

The first principal component about the mean of a set of points can be represented by that line which most closely approaches the data points (as measured by squared distance of closest approach, i.e. perpendicular to the line). In contrast, linear least squares tries to minimize the distance in the $y$ direction only. Thus, although the two use a similar error metric, linear least squares is a method that treats one dimension of the data preferentially, while PCA treats all dimensions equally.

Regularized versions

Main article: Regularized least squares

Tikhonov regularization

Main article: Tikhonov regularization

In some contexts a regularized version of the least squares solution may be preferable. Tikhonov regularization (or ridge regression) adds a constraint that $\|\beta \|^{2}$, the L₂-norm of the parameter vector, is not greater than a given value. Equivalently, it may solve an unconstrained minimization of the least-squares penalty with $\alpha \|\beta \|^{2}$ added, where $\alpha$ is a constant (this is the Lagrangian form of the constrained problem). In a Bayesian context, this is equivalent to placing a zero-mean normally distributed prior on the parameter vector.

Lasso method

An alternative regularized version of least squares is Lasso (least absolute shrinkage and selection operator), which uses the constraint that $\|\beta \|$, the L₁-norm of the parameter vector, is no greater than a given value. (As above, this is equivalent to an unconstrained minimization of the least-squares penalty with $\alpha \|\beta \|$ added.) In a Bayesian context, this is equivalent to placing a zero-mean Laplace prior distribution on the parameter vector. The optimization problem may be solved using quadratic programming or more general convex optimization methods, as well as by specific algorithms such as the least angle regression algorithm.

One of the prime differences between Lasso and ridge regression is that in ridge regression, as the penalty is increased, all parameters are reduced while still remaining non-zero, while in Lasso, increasing the penalty will cause more and more of the parameters to be driven to zero. This is an advantage of Lasso over ridge regression, as driving parameters to zero deselects the features from the regression. Thus, Lasso automatically selects more relevant features and discards the others, whereas Ridge regression never fully discards any features. Some feature selection techniques are developed based on the LASSO including Bolasso which bootstraps samples, and FeaLect which analyzes the regression coefficients corresponding to different values of $\alpha$ to score all the features.

The L¹-regularized formulation is useful in some contexts due to its tendency to prefer solutions where more parameters are zero, which gives solutions that depend on fewer variables. For this reason, the Lasso and its variants are fundamental to the field of compressed sensing. An extension of this approach is elastic net regularization.

See also

• Adjustment of observations
• Bayesian MMSE estimator
• Best linear unbiased estimator (BLUE)
• Best linear unbiased prediction (BLUP)
• Gauss–Markov theorem
• L₂ norm
• Least absolute deviation
• Measurement uncertainty
• Orthogonal projection
• Proximal gradient methods for learning
• Quadratic loss function
• Root mean square
• Squared deviations

Further reading

• Björck, Å. (1996). Numerical Methods for Least Squares Problems. SIAM. ISBN 0-89871-360-9. 
• Kariya, T.; Kurata, H. (2004). Generalized Least Squares. Hoboken: Wiley. ISBN 0-470-86697-7. 
• Luenberger, D. G. (1997) [1969]. "Least-Squares Estimation". Optimization by Vector Space Methods. New York: John Wiley & Sons. pp. 78–102. ISBN 0-471-18117-X. 
• Rao, C. R.; Toutenburg, H.; et al. (2008). Linear Models: Least Squares and Alternatives. Springer Series in Statistics (3rd ed.). Berlin: Springer. ISBN 978-3-540-74226-5. 
• Wolberg, J. (2005). Data Analysis Using the Method of Least Squares: Extracting the Most Information from Experiments. Berlin: Springer. ISBN 3-540-25674-1. 

UpToDate Contents

全文を閲覧するには購読必要です。 To read the full text you will need to subscribe.

• 1. 乳幼児および思春期前の小児における通常の成長パターン normal growth patterns in infants and prepubertal children
• 2. 在胎月齢の生後評価 postnatal assessment of gestational age
• 3. Fertility awareness法での避妊 fertility awareness based methods of pregnancy prevention
• 4. 低身長の小児および思春期患者に対する診断的アプローチ diagnostic approach to children and adolescents with short stature
• 5. 腰痛の運動療法 exercise based therapy for low back pain

English Journal

• Determination of trace thiophanate-methyl and its metabolite carbendazim with teratogenic risk in red bell pepper (Capsicumannuum L.) by surface-enhanced Raman imaging technique.

• Li JL1, Sun DW2, Pu H1, Jayas DS3.
• Food chemistry.Food Chem.2017 Mar 1;218:543-552. doi: 10.1016/j.foodchem.2016.09.051. Epub 2016 Sep 13.
• Surface-enhanced Raman scattering (SERS) imaging coupling with multivariate analysis in spectral region of 200 to 1800cm-1 was developed to quantify and visualize thiophanate-methyl (TM) and its metabolite carbendazim residues in red bell pepper (Capsicum annuum L.). Least squares support vector mac
• PMID 27719947

• How to predict the sugariness and hardness of melons: A near-infrared hyperspectral imaging method.

• Sun M1, Zhang D1, Liu L2, Wang Z3.
• Food chemistry.Food Chem.2017 Mar 1;218:413-421. doi: 10.1016/j.foodchem.2016.09.023. Epub 2016 Sep 7.
• Hyperspectral imaging (HSI) in the near-infrared (NIR) region (900-1700nm) was used for non-intrusive quality measurements (of sweetness and texture) in melons. First, HSI data from melon samples were acquired to extract the spectral signatures. The corresponding sample sweetness and hardness values
• PMID 27719929

• Varietal classification and antioxidant activity prediction of Osmanthus fragrans Lour. flowers using UPLC-PDA/QTOF-MS and multivariable analysis.

• Zhou F1, Peng J1, Zhao Y1, Huang W2, Jiang Y1, Li M1, Wu X3, Lu B4.
• Food chemistry.Food Chem.2017 Feb 15;217:490-7. doi: 10.1016/j.foodchem.2016.08.125. Epub 2016 Aug 31.
• This study was aimed to classify the varieties and predict the antioxidant activity of Osmanthus fragrans flowers by UPLC-PDA/QTOF-MS and multivariable analysis. The PLS-DA model successfully classified the four varieties based on both the 21 identified compounds and the effective compounds. For the
• PMID 27664663

Japanese Journal

• 最小二乗法による中国古代星図の年代推定

• 竹迫 忍
• 数学史研究 = Journal of history of mathematics, Japan (232), 1-21, 2019-12
• NAID 40021840267

• 機械学習を活用したアーク溶接部特性に関する実験式の自動導出 (特集 深層学習・機械学習を応用した溶接技術)

• 北野 萌一,中村 照美
• 溶接学会誌 = Journal of the Japan Welding Society 88(7), 532-535, 2019-10
• NAID 40022026695

• 衛星アンテナパターン計測のための二次元最小二乗法による再構成手法に関する検討 (マイクロ波)

• 藤野 義之,高橋 完次,松戸 太一
• 電子情報通信学会技術研究報告 = IEICE technical report : 信学技報 119(12), 27-32, 2019-04-22
• NAID 40021885288

Related Links

• 最小二乗法(least squares method)

最小二乗法(least squares method) 前回は線形回帰とは、y = ax + b のa、bを求めることで未知のxに対し、yを予想できるようになることと書きましたが、 そのa、bを求めるための方法が最小二乗法になります。 最小二乗法について勉強した ...

• Least Squares Method Definition - Investopedia

The least squares method provides the overall rationale for the placement of the line of best fit among the data points being studied. The most common application of the least squares method, which is referred to as linear ...

• The Method of Least Squares - Williams College

The Method of Least Squares Steven J. Miller⁄ Mathematics Department Brown University Providence, RI 02912 Abstract The Method of Least Squares is a procedure to determine the best fit line to data; the proof uses simple ...

Related Pictures

---

★リンクテーブル★

リンク元	「最小二乗法」「least-squares analysis」
関連記事	「method」「square」「least squares」「least」

「最小二乗法」

　　[★]

英

least-squares analysis least-squares method least square method

関

最小自乗、リートベルト法

「least-squares analysis」

　　[★]

最小二乗法

関

least squares、least-squares method、Rietveld refinement

「method」

　　[★]

• n.

• 方法、手法、測定法、測定方法、方式

関

assay、fashion、law、manner、means、method for measurement、-metry、mode、procedure、process、system、technique、way

　 　

「square」

　　[★]

• n.

• 平方、二乗、四角

• adj.

• 四角の

関

sq

　 　 　 　 　 　 　 　 　 　 　 　 　 　 　 　 　 　

「least squares」

　　[★]

最小二乗、最小自乗

関

least-squares analysis、Rietveld refinement

「least」

　　[★]

• little