By Matt Prodger
BBC Newsnight, Yemen
But for Amr Khaled he was willing to make an exception. The Egyptian Muslim televangelist is, in the words of the president a "good guy", but there is of course more to it than that - in the Middle East Amr Khaled is like a rock star.
My big aim is to uproot extremism in Yemen by encouraging people to be positive, face down the extremists and say 'We don't want you in our country
He is a sharp-suited, smooth-talking whirlwind of progressive Islam, preaching co-existence with the West while telling Muslims how to mix their faith with the modern world.
He is not even an imam, but a former accountant for multinational auditors KPMG.
It is easy to see why Mr Khaled's massive popularity has unnerved largely authoritarian governments in the Middle East - in the past he has been forced to leave his native Egypt and broadcast from the UK - he is calling for social change, of sorts:
"We need a better future in the Middle East. We need development in this area of the world. Youth in this area need a chance and the world should listen to them, support them," he told me when we met in Yemen.
Mr Khaled is visiting Yemen on an ambitious mission to, as he puts it, take the battle against al-Qaeda to its heartland.
I also went to the presidential palace in the port city of Aden, where Mr Saleh granted Newsnight an interview, a rarity for Western media.
Mr Saleh insisted that far from being a cradle of terror, "Yemen is a victim of terrorism".
When challenged about why he himself has failed to kick al-Qaeda out of the country he threw the question back, asking:
"Why should Yemen be expected to get rid of terrorism before others have? Why hasn't the United States and its allies got rid of terrorism in Afghanistan, Pakistan and Iraq?"
"Yemen is carrying out its own efforts through its army and security apparatus to fight terrorism and has scored excellent victories. So Yemen is not a cradle for terrorism, on the contrary it rejects it, is fighting it, and in as much has lost tens of victims from the police, army and citizens."
But recently this claim of Yemeni victories has been dented by documents from whistleblower website, Wikileaks.
He is not like other clerics with beards. He talks to youths without boundaries and that is why he is so attractive to me."
Young Yemeni woman
The reason? The Yemeni government does not want to look like it is kowtowing to the US by granting unrestricted access.
In many ways Yemen is ideal territory for al-Qaeda. It is a tribal society with limited central government control.
A civil war in the north has been going on for so long that few recall what started it, there is rampant corruption and in the south a separatist struggle.
Yemeni journalist Zaid Ali al-Alaya'a says al-Qaeda has also benefitted from a "lack of trust and co-operation" between Saudi, Yemeni and Western intelligence organisations.
Message of moderation
Mr Khaled's mission to end al-Qaeda's grip began with a media blitz, and a speech at the enormous Saleh Mosque in the capital Sanaa.
Addressing an audience of fifty thousand people inside the mosque, several thousand more outside, and millions more Yemenis watching live on TV, his message was uncompromising - it is your religious duty to safeguard moderation.
"Every father, every mother, must shield their children from the creeping reach of extremism," he urged.
Mr Khaled's plan to change Yemen centres on young people. "Youth can be the difference, youth can change, can take out the roots of the extremism in Yemen," he told me.
In concrete terms his Right Start organisation has begun training youth leaders and imams to take his message of moderation into mosques and schools.
One man who understands the lure of al-Qaeda is Nasser al-Bahri, Osama Bin Laden's former bodyguard and a veteran of Jihadi missions in Bosnia, Somalia and Afghanistan.
When I met Bahri, who says he is now a reformed character, he voiced concern about the younger generation of extremists in Yemen:
"This latest generation is motivated by anger with the government" he said. "This is very dangerous because most of south Yemen wants independence, so their ideas are mixed up with that struggle."
"It's more difficult to reason with them, and they're more ignorant than previous generations," he added.
Bahri said that while he supports the aims of moderate Muslims like Amr Khaled, al-Qaeda is so determined to draw US soldiers into conflict in Yemen, that the likes of Mr Khaled are unlikely to be able to stop them:
"From what I have seen they absolutely do not affect in any manner whatsoever the programme or work of al-Qaeda," he said.
Critics say US counter-terrorism funds would be better spent on alleviating poverty
"He is not like other clerics with beards," one told me. "He talks to youths without boundaries and that is why he is so attractive to me."
"I think he is trying to solve the crisis, but he is trying to make it a bit easier and more fun along the way," said another. "So I think he is doing it in a very good way, but we have to wait for the results."
But it will take more than sermons to uproot al-Qaeda. The US is pouring hundreds of millions of dollars into Yemen in a bid to combat the terror network, but that money is ring-fenced for counter-terrorism measures.
Critics say the money would be better spent on alleviating poverty and addressing the resentment that recruits for al-Qaeda.
Yemen is hooked on a dwindling oil supply for three quarters of its revenue, it is running out of water, and a third of its adults are out of work.
As one man said to me: "Al-Qaeda is your problem, we've got bigger ones."