Yes, main points brought out by Richard and Kort above.  The lack of focusing motor is one of the biggest knocks against the D40 series.

Some other areas to note;
The D40 does not have a top status LCD monitor.
The D40 does not have depth of field preview.
If I recall correctly the D40 does not have bracketing capability
The D40 uses a less sophisticated (slower) focus aquisition...and I think it has less focus areas as well.

Physically, the D40 is a much smaller camera and in essence a "stripped" down version of the D80.  You will have to decide for yourself whether the additional features of the D80 are necessary (and worth the extra cost) for your shooting situations.
The comparison link given by Kort should give you a good start to review the technical differences...but don't forget to handle each cam physically before you purchase.

Unless you already have Nikon glass...you may want to consider the other brands as well...

In youth, we learn....with age, we understand.