Portion size affects intake, but when all foods are served in large portions, it is unclear whether every food will be consumed in greater amounts. We varied the portion size (PS) of all foods at a meal to investigate the influence of food energy density (ED) on the PS effect as well as that of palatability and subject characteristics. In a crossover design, 48 women ate lunch in the laboratory on four occasions. The meal had three medium-ED foods (pasta, bread, cake) and three low-ED foods (broccoli, tomatoes, grapes), which were simultaneously varied in PS across meals (100%, 133%, 167%, or 200% of baseline amounts). The results showed that the effect of PS on the weight of food consumed did not differ between medium-ED and low-ED foods (p < 0.0001). Energy intake, however, was substantially affected by food ED across all portions served, with medium-ED foods contributing 86% of energy. Doubling the portions of all foods increased meal energy intake by a mean (±SEM) of 900 ± 117 kJ (215 ± 28 kcal; 34%). As portions were increased, subjects consumed a smaller proportion of the amount served; this response was characterized by a quadratic curve. The strongest predictor of the weight of food consumed was the weight of food served, both for the entire meal (p < 0.0001) and for individual foods (p = 0.014); subject characteristics explained less variability. Intake in response to larger portions was greater for foods that subjects ranked higher in taste (p < 0.0001); rankings were not related to food ED. This study demonstrates the complexity of the PS effect. While the response to PS can vary between individuals, the effect depends primarily on the amounts of foods offered and their palatability compared to other available foods.