Not only do high-income taxpayers have more complicated returns, but they bring in much more revenue for the IRS with each mistake they make.

While only 1% of the overall population gets audited, the odds jump to 21% for taxpayers with income over $5 million and to 30% for those earning $10 million or more, according to the most recent statistics from the IRS.

"It's not that higher income taxpayers cheat more, it's just that you have a lot more going on on a high-income return," said Betsey Buckingham, an enrolled agent at accounting firm David C. Murray & Company. "Most of the high-income people I've [assisted] are involved in charities or very active in their own business."

Even if you're not rich but live in a wealthy neighborhood, your return could raise questions about how you can afford to live there -- especially if you report surprisingly low income or a big business loss.

"They notice if you don't have an income that closely matches the kind of lifestyle you live," said Buckingham.

NEXT: You say the wrong things