Richard,
If you intended to continue working for a long time having the mortgages might be considered a kind of "businessman's risk"--- you'd have leveraged your financial assets of $1.2 million with $405k of debt. That's a lot, but it's easily managed because of your $50k of labor earnings.
Deciding not to work makes the debt proposition a lot more risky for several reasons. One is that the interest rate on your mortgages is probably a good deal greater than you are earning on the $320k you have in cash and bonds and it's probably greater than the return you're earning on your $100k variable annuity investment.
So why bother leveraging? Pay off the mortgages and your cash flow requirements will decline and you won't be facing any chance that your mortgage interest rate will rise quickly, requiring more cash just as the value of your fixed income investments is tanking.
Depending on how much you spend to support your standard of living, you might also enjoy some reduction in the taxation of your Social Security benefits.
Scott
