A study of 8,800 people over 50 showed high blood pressure and being overweight also seemed to affect the brain, but to a lesser extent.
Scientists involved said people needed to be aware that lifestyles could damage the mind as well as the body.
Their study was published in the journal Age and Ageing.
Researchers at King’s were investigating links between the likelihood of a heart attack or stroke and the state of the brain.
Data about the health and lifestyle of a group of over-50s was collected and brain tests, such as making participants learn new words or name as many animals as they could in a minute, were also performed.
They were all tested again after four and then eight years.
The results showed that the overall risk of a heart attack or stroke was “significantly associated with cognitive decline” with those at the highest risk showing the greatest decline.
It also said there was a “consistent association” between smoking and lower scores in the tests.
One of the researchers, Dr Alex Dregan, said: “Cognitive decline becomes more common with ageing and for an increasing number of people interferes with daily functioning and well-being.
“We have identified a number of risk factors which could be associated with accelerated cognitive decline, all of which, could be modifiable.”
Dr Simon Ridley, from Alzheimer’s Research UK, said: “Research has repeatedly linked smoking and high blood pressure to a greater risk of cognitive decline and dementia, and this study adds further weight to that evidence.
“Cognitive decline as we age can develop into dementia, and unravelling the factors that are linked to this decline could be crucial for finding ways to prevent the condition.
“These results underline the importance of looking after your cardiovascular health from mid-life.”
The Alzheimer’s Society said: “We all know smoking, a high blood pressure, high cholesterol levels and a high BMI [Body Mass Index] is bad for our heart. This research adds to the huge amount of evidence that also suggests they can be bad for our head too.
“One in three people over 65 will develop dementia but there are things people can do to reduce their risk. – BBC