You should do this. But you need to do it very carefully. Let me explain why.
The most important fact here is that even a 5.1 percent interest rate is hard to get in today's market without taking significant risk. So you're probably NOT earning 5.1 percent on the savings that would go to paying off the mortgage.
Your mortgage also has principal payments and those have to be taken from a source like your IRA account, increasing your taxable income.
As you get older your RMD (Required Minimum Distribution) from your IRA will get larger and larger, increasing your taxes and increasing the odds that you will have to pay taxes on a portion of your Social Security benefits. You may benefit from paying the mortgage off now or over the next two years.
The hard part is finding a way to pay off the mortgage without creating an undue tax burden on yourself. For a joint return in 2008 you can have taxable income up to $65,100 and remain in the 15 percent tax bracket. Since you also have a $10,800 standard deduction, a $2,100 elderly deduction, and $7,000 in personal exemptions, you can have another $19,900 in income before jumping into the 25 percent tax bracket. That's a total income of $85,000 before you jump brackets.
That tells me that you should pay part of your mortgage off in December and pay off the remainder in January, so you'll avoid paying taxes at an extra-high rate just because you're paying debt off. You can, and should, get the exact numbers worked out by a good tax accountant.
Scott