Too little of one type of cholesterol has been linked by research to memory loss and Alzheimer's disease.
UK and French scientists studied 3,673 civil servants, revealing low levels of "good" cholesterol were associated with poor memory.
Doctors might be able to uncover high-risk patients using blood tests, they said in a US heart journal.
But other experts said the study did not yet support larger diet trials aiming to boost levels.
The relationship between levels of HDL, or "good", and LDL, or "bad" types of cholesterol is thought to be important in the development of other serious conditions such as heart disease and stroke.
Higher levels of HDL, in particular, are believed to protect against damage to blood supply caused by the narrowing of the arteries.
There is also evidence that "good" cholesterol can influence the laying down of the beta-amyloid "plaques" that are a distinctive feature in the brains of Alzheimer's patients.
Regular exercise and eating less saturated fat, while eating more "healthy" fats such as olive oils, can boost levels.
The researchers, from University College London and the INSERM institute in France, used data from the Whitehall II trial - a collection of thousands of civil servants, to see what influence it might have over memory within a five-year period.
They took blood samples at the start of the study, and gauged word recall with a simple test. That was repeated again at the end of the study.
The researchers found that people with low levels of HDL were 53% more likely to suffer memory loss compared with the people with the highest levels of HDL.
People with impaired memory have a much greater risk of going on to develop dementia later in life.
Dr Archarna Singh-Manoux, who led the study, said: "Memory problems are key in the diagnosis of dementia.
"This suggests that low HDL cholesterol might also be a risk factor for dementia."
She said that doctors should be encouraged to monitor HDL levels in order to predict dementia risk.
However, an editorial in the same journal, Arteriosclerosis, Thrombosis, and Vascular Biology, by Dr Anatol Kontush and Dr John Chapman, from INSERM and the Universite Pierre and Marie Curie in Paris, said that the study did not prove that low HDL could cause memory loss, or high HDL protect against it.
"Unfortunate results in large interventional trials with dietary antioxidants suggest that we should remain cautious when proposing therapeutic intervention," they wrote.
Dr Susanne Sorenson, from the Alzheimer's Society, said HDL cholesterol was believed to transport harmful cholesterol from the arteries back to the liver to be degraded.
"This study shows that if there is not enough HDL to transport cholesterol and other lipids around the body, it can not only increase your risk of heart disease but also affect your memory and may increase your risk of getting Alzheimer's disease.
"We know that controlling cholesterol in midlife is important if you are to reduce your risk of developing vascular dementia later and this may also be important for the development of Alzheimer's disease."