Tuesday, August 28, 2012

For Dummies: Intro to Generalized Method of Moments


While researching various concepts from statistics and econometrics I constantly find, that many of the materials describing the concepts I'm researching are often too obscure.

Before I get through history, definitions, derivations of formulas, proofs and lemmas to applications and examples - things that I'm really interested in when I'm just starting to research something - I'm usually too bored to care... :(

In addition, many authors of the research papers and article assume, the reader knows everything the author does, and sometimes some authors are simply reluctant to reveal all the details of their approach. As a result, descriptions of the methods or experiments sometimes lack crucial parts, while the examples are not very informative for being as unclear as the description of the method itself.

Hence, I decided to write a couple of posts that will try to present some concepts in layman's terms.

Hope, you will enjoy :)

*   *   *

Generalized Method of Moments (GMM)  is a method of estimating parameters of a probability distribution (such as mean and standard deviation in the case of normal distribution), by checking what possible values of distribution parameters lead to the best fitting moments of the sample drawn from the distribution.

Moments are some measures describing the shape of the distribution. In case of the normal distribution, the four first moments are:

1. mean
2. variance -> standard deviation
3. skewness
4. kurtosis

So let's say, we have n=100 observations x drawn from the normal distribution which has some unknown parameters:

x ~ N(mean, standard deviation)

> x <- rnorm(n,x.mean,x.sd)

Fig. Density of the sample

We would like to find out, what these parameters are.

Since, we are dealing with the normal distribution, it is very simple to find mean and standard deviation of the sample:

> mean(x)
[1] -7.176239

> sd(x)
[1] 6.105964

However, the sample parameters may not equal the distribution parameters.

Here we can use the before mentioned Generalized Method of Moments. It uses not only mean and variance (-> standard deviation), but also other moments of the distribution, to find better parameters.

You can use the sample estimates as starting points in searching for distribution parameters:

> t0 <- c(mean(x),sd(x)) # starting points <- sample estimates

> coef(gmm(g2,x,t0)) # GMM 2 moments

 Theta[1]  Theta[2] 
-7.176255  6.075357 

> coef(gmm(g3,x,t0)) # GMM 3 moments
 Theta[1]  Theta[2] 
-7.099134  5.981886 

> coef(gmm(g4,x,t0)) # GMM 4 moments
 Theta[1]  Theta[2] 
-6.995580  5.838024

In this case the more moments GMM uses, the value of mean increases while standard deviation decreases. As a results, the parameters estimated by GMM are getting closer to the true values of the distribution parameters, which are:

> x.mean
[1] -6.701526

> x.sd
[1] 5.493381

And that's Generalized Method of Moments in a nutshell ;)

BTW: You can alternatively use the Maximum Likelihood Method for the same task.

The results are very similar to GMM with 2 moments:

> o <- optim(c(0,1),objFun)

> o$par
[1] -7.176543  6.072980

But that's a totally different story :)

GMM portion of the R code based on Pierre Chausse, "Computing Generalized Method of Moments and Generalized Empirical Likelihood with R"

[ R source ]