Height was also affected by the prevalence of disease

Both endemic and epidemic diseases were far more prevalent in the past. Disease affected the ability of people to effectively find food and to process its nutritional content.

To come back to your point on nobility. They would have increased access to food, which is likely to have meant they they were in indeed larger than their peasant counterparts within the same time period. However they would still have been affected by a myriad of diseases (viral, bacterial and parasitic), which is likely to have stunted their growth compared to modern day humans.

Because a mother's nutrition-induced smallness limits her offspring's height

Height is affected not only by one's own nutrition but also by one's mother's nutrition. Small mothers are more likely to produce small fetuses who are more likely to be small adults.

For example: a woman has a gene which means that, with good nutrition, she should be very tall. However she grows up during a famine and is therefore small. Because she is small, all her fetuses are relatively small (regardless of their genes).

So basically, the purely non-genetic, environmental factors affecting your mother, also affect you. This means that it takes generations for people to reach the full height which their genes give them the potential for.

Nutrition is the biggest part of the story

Current knowledge indicates that better access to the range of nutrients that humans require to function and grow effectively has had the biggest impact on height. There are also other factors: for example, studies tried to determine why the Netherlands has the highest average heights globally - the conclusion was superior neo-natal care.