We use the Survey of Consumer Finances (SCF) from 1995 through 2016 to study trends in average effective tax rates across the income and wealth distribution. These average tax rates (ATRs) calculated from SCF income data are comparable to those calculated from external sources. We show that the wealthiest families have the highest ATRs, even as the income definition expands to include nontaxable sources and even though the wealthiest families are only sometimes among those with the highest annual income. However, the majority of income-producing assets held by the wealthiest are in the form of unrealized capital gains, effectively avoiding taxation. After altering the income concept to a measure of “potential income,” which incorporates changes in net worth and allows us to include untaxed increases in asset values, the wealthiest families no longer have the highest ATRs.