A Serious Challenge to the 2012 Low-carbohydrate "Metabolic Advantage" Study

Warning: this post will be a bit more wonkish than usual, because I need to get detailed to make my points.  To read a summary, skip to the end.

In 2012, David Ludwig's group published an interesting RCT that suggested a substantial "metabolic advantage" resulting from a high-protein, very-low-carbohydrate diet (VLC) (1).  In other words, this diet led to a higher energy expenditure relative to a normal-protein, low-fat diet (LF) over a one month period (a low-glycemic-load, normal-protein diet was in the middle and not significantly different from the other two).  Resting energy expenditure (REE) was slightly but significantly higher on the VLC diet, and total energy expenditure (TEE) was elevated by a whopping 300+ kcal/day!  I covered the study at the time, describing it as "fascinating" and "groundbreaking", and calling for the study to be replicated so we can be more confident in its unexpected result (2).

This finding has been used by Ludwig, Gary Taubes, and others to support the carbohydrate-insulin hypothesis of obesity, although there is no evidence that the effect was mediated by insulin, and also no evidence that it was mediated by reduced carbohydrate rather than increased protein (3).

Since I published that post, my confidence in the finding-- and particularly the common interpretation of it that reducing carbohydrate intake to a very low level increases REE and TEE-- has gradually been eroding.  This is partially because other studies have generally reported that the carbohydrate:fat ratio of the diet has little or no effect on REE, TEE, or fat storage (4, 5, 6, 7, 8, 9, 10).

My confidence level dropped even more this week, when I saw a critical comment Kevin Hall added to the study in PubMed Commons (11).  Ludwig and colleagues have since responded (12).  Here's my perspective on the exchange, organized by topic:

Inconsistency between reported energy expenditure and weight change

Hall pointed out that measured TEE was ~200-500 kcal/day higher than reported energy intake for all diets, with the largest gap in the VLC diet.  In other words, 200-500 more kilocalories were going out than were going in.  Over a four-week period, if the data are correct, the VLC diet should have caused just over a kilogram of weight loss-- but it didn't-- the VLC diet group actually gained a small amount of weight (0.5 kg).  This internal inconsistency in the data may suggest that one or more of their measures is incorrect, either body weight (unlikely), energy intake (likely, as explained below), and/or energy expenditure (possible, as explained below).

Ludwig responded that body weight is an unreliable measure of energy balance over short periods of time-- in other words, there could have been differential changes in body composition between groups that masked a body weight loss effect.  I do agree that small changes in body weight over short periods of time aren't a very solid measure of energy balance, but what Ludwig is proposing here is basically that the VLC group was gaining water weight and/or lean mass to offset fat loss that was simultaneously occurring.  This hypothesized water/lean mas gain would have to offset more than 1.5 kg of weight change (just over 1 kg predicted from TEE, + 0.5 kg weight gain on VLC diet).  Since we know that VLC diets tend to reduce water weight, and are unlikely to pack on 1.5+ kg of lean mass in a month (particularly in the face of a diet that does not meet energy needs), Ludwig's explanation is hard to believe.

This brings us to what is probably the real reason for the lack of change in weight.

Incomplete diet adherence

When I first wrote about this study, I don't think I fully appreciated the likelihood that the volunteers would "cheat" and eat outside food.  Ludwig's team provided them with all study foods, but since it wasn't a metabolic ward study, volunteers had the ability to eat additional food at home.  I'll add that these people had a strong incentive to eat more, because they were maintaining a 10-15% reduced body weight throughout the study.  This means they were hungry and highly motivated to eat additional food at home.

And they probably did.  This may explain the fact that the volunteers gained a bit of weight on all diets, despite the fact that their supposed energy intake was hundreds of kilocalories lower than their energy expenditure!  Ludwig acknowledges this in his comment, saying "it is likely that some non-study foods were consumed".  He downplays it, but we're talking about 200-500+ kcal/day here, which is not insignificant.

I understand that this kind of study is challenging to do, and the design is certainly a lot more rigorous than the type of diet study where you just send people home with diet advice and let them cook their own food.  This type of study design works well in a lot of contexts, but perhaps it isn't the best suited for situations where your volunteers are weight-reduced and have a strong incentive to eat additional food.

Statistical weaknesses

The study declared REE as the primary outcome, and it found that the VLC diet had a significantly higher REE than the LF diet.  The difference was only 67 kcal/day, but this finding is robust because it was declared in advance as the primary outcome.  I think we can be fairly confident that, under the conditions tested, the VLC diet led to an increase in REE vs. the LF diet.  However, the increase is small enough that it has little clinical or practical significance.

But the big splash made by this study wasn't the 67 kcal/day increase in REE-- it was the ~300 kcal/day increase in TEE!  Although this finding got all the press, it rests on much less solid ground, as pointed out by Hall.  The reason is that it's one of many secondary outcomes (66 possible statistical tests performed on 22 measured variables, to be exact), and when you have so many possible outcomes, you end up with a high likelihood of a false positive finding unless you use specific statistical tools to correct the problem.  This is called the multiple comparisons problem and it's very common in science.  In fact, it's one of the leading causes of unreliable findings in the scientific literature.

As I said, the multiple comparisons problem can be avoided by using the right statistical tools (essentially, raising the bar for statistical significance in proportion to the number of statistical tests you perform), but in Ludwig's study, these tools were not sufficiently applied to give us confidence in the secondary outcomes.  And that's not actually as bad as it sounds, but it does necessitate an additional layer of interpretation.  Basically, if you don't adequately correct for multiple comparisons, your finding is "exploratory"-- meaning it's not a statistically robust finding, but it is suggestive and can be used to guide further research.  Researchers do this all the time, and in my opinion it's acceptable as long as it's clearly labeled as exploratory.  But it also means that you can't (or shouldn't) proclaim to the public that a diet increases TEE by hundreds of kilocalories per day, because that result is not statistically robust.

So to summarize, we're left with a small REE effect that is statistically robust, and a large TEE effect that is statistically questionable.  And as we will see, the REE effect is likely explained by the higher protein content of the VLC diet.

The confounding effect of protein

The VLC diet was 30% protein by calories, while the other two diets were 20% protein.  Since high-protein diets are known to increase energy expenditure, this may provide a plausible explanation for the observed increase in REE on the VLC diet.

In Ludwig's response to Hall's critique, Ludwig states that the higher protein content of the VLC diet can't explain the higher REE: "A protein difference of this magnitude can’t explain differences in REE in the fasting state, long after the thermic effects of food have dissipated".  But in fact, other RCTs demonstrate that it can explain the increase in REE.

In 2013, the research group of Margriet Westerterp-Plantenga at Maastricht University-- one of the leading researchers in this area-- published a study that examined the effects of high-protein vs. normal-protein diets on REE after weight loss (13).  It's worth noting that protein was increased at the expense of fat, not carbohydrate.

Following a weight loss of about 7 kg in both groups, the normal-protein group experienced a reduction of REE, but the high-protein group didn't.  The between-group difference in REE during weight maintenance?  81 kcal/day.  This is very similar to the 67 kcal/day that Ludwig reported, except that it was produced by increasing protein alone, not a combination of high protein and very low carbohydrate as in Ludwig's study.  In my view, high protein intake probably explains the elevated REE in the VLC diet group.

Summary


The 2012 study by David Ludwig's group reported that in subjects maintaining weight loss, a high-protein, very-low-carbohydrate diet increased resting energy expenditure by a small amount (67 kcal/day), and total energy expenditure by a large amount (~300 kcal/day), relative to a normal-protein, low-fat diet.  Superficially, this seems to support the idea of a substantial "metabolic advantage" for very-low-carbohydrate diets.  However, there are reasons to be skeptical of some of the findings, which Kevin Hall laid out in a comment on NIH Commons.  Here's a summary of my current views:

• There was probably significant dietary non-compliance, with volunteers eating extra food at home in addition to what they were supplied by the lab.  This probably explains at least part of why they didn't lose weight, despite the fact that they should have lost weight according to the reported energy intake and expenditure figures.


• The finding that resting energy expenditure was elevated on the VLC diet is statistically robust.  However, the increase is small enough that it has little clinical or practical significance.


• The increase in resting energy expenditure can be explained by the higher protein content of the VLC diet, and it probably had little to do with the fact that the VLC diet was low in carbohydrate.


• The finding that total energy expenditure was elevated by ~300 kcal/day on the VLC diet is not statistically robust, and it's also hard to reconcile with the results of other studies.  

For me, here's the bottom line.  Aspects of the study still stand as reliable, but the findings do not support the conclusion that reducing carbohydrate intake yields a metabolic advantage.  My best interpretation is that the study is consistent with other findings suggesting that high-protein diets yield a small metabolic advantage, but that calorie intake is still the key determinant of fat mass changes.