Cross-sectional and longitudinal studies present association of low dietary energy density with higher intake of vitamins, minerals and dietary fiber, lower intake of fat, and better balance of macronutrients. The objective of this study was to verify the relationship between dietary energy density and diet quality measured by an index of diet quality. This study used data from 496 adults and 445 older adults of cross-sectional population-based survey from São Paulo conducted in 2008–2009, Brazil. Dietary intake data was assessed by two 24-h dietary recalls. Dietary energy density values were calculated based on foods only method. Dietary energy density and revised Brazilian Health Eating Index and its components, were estimated by usual intake using Multiple Source Method. The relationship between dietary energy density and the total revised Brazilian Health Eating Index and its components were assessed by Gaussian family log-link model for each age group. The analyses showed an inverse association between dietary energy density and total revised Brazilian Health Eating Index in adults (T2:β = 0.96, p < 0.001; T2:β = 0.86, p < 0.001) and older adults (T2:β = 0.96, p < 0.001; T2:β = 0.90, p < 0.001), and an inverse association between dietary energy density and nine of twelve revised Brazilian Health Eating Index components in adult and/or older adults groups. Dietary energy density was associated with diet quality in Brazilian adults and older adults regardless of sex, per capita household income, body mass index, physical activity level, current smoking habits status, alcohol beverage drinking status and usual energy intake (kilocalories) from beverages.