Kenya's police chief David Kimaiyo stood down, while Interior Minister Joseph Ole Lenku was dismissed.
Earlier, al-Shabab killed 36 quarry workers in the north-eastern Mandera region near the Somali border.
The group attacked the workers around midnight on Monday while they were asleep in tents at the quarry in Kormey, 15km (nine miles) from the town of Mandera.
Non-Muslim workers were shot dead after being separated from the Muslims.
'Choose a side' "This is a war against Kenya and Kenyans," Mr Kenyatta said on national TV on Monday. "It is a war that every one of us must fight."
"The time has come for each and every one of us to decide and choose - are you on the side of an open, free, democratic Kenya... or do you stand with repressive, intolerant extremists?"
He said Interior Minister Joseph Ole Lenku had been fired, and he nominated an opposition politician and former army general, Joseph Nkaissery, as his replacement.
The president also announced that he had accepted Mr Kimaiyo's wish to retire.
Correspondents say both Mr Kimaiyo and Mr Lenku have been under pressure to resign amid growing concern over security in Kenya following a spate of attacks.
Analysis: Robert Kiptoo, BBC Africa, Nairobi Most Kenyans will be pleased by the departure of police chief David Kimaiyo and Interior Minister Joseph Ole Lenku - two men widely blamed for the failure to get to grips with the insurgency.
President Kenyatta has shown his determination to declare war on al-Shabab by nominating a former army general as the new interior minister.
If Kenya's parliament approves his nomination, Joseph Nkaissery will become the first opposition MP handed such a key ministerial post since Kenya adopted a new constitution in 2010.
Like his predecessor, Mr Nkaissery comes from the Maasai community, suggesting the president took into account the need to ensure the ethnic group remains represented in government.
At a time when al-Shabab is threatening Kenya's security, Mr Kenyatta cannot afford to cause ethnic tension by alienating any group.
A driver who visited the scene of Monday's attack, Ali Sheikh Yusuf, told the BBC most of the victims appeared to have been lined up and shot in the head at close range.
He said four were beheaded inside their tents, while three appeared to have escaped to Mandera town.
Al-Shabab said it carried out the attack, blaming the involvement of Kenyan forces in Somalia "and their ongoing atrocities therein, such as the recent air strikes on Muslims".
The group put the number of those killed at 40, higher than official accounts.
Kenya's Red Cross said that security personnel and one of its own teams were at the scene soon after the attack.
Al-Shabab is based in Somalia but has stepped up its campaign in Kenya since 2011, when Kenya sent troops across the border to help battle the militants.
Only last week the Islamist group killed 28 people in Mandera county in an attack on a bus targeting non-Muslims.
Hundreds of people later fled to a military strip, demanding the government evacuate them from the region.
Demonstrators took to the streets in the Kenyan capital, Nairobi, calling on the president to improve security.
Dozens of people have also been killed in a series of shootings in coastal districts in recent months.
In one of the worst attacks on Kenyan soil, 67 people were killed in September last year when four gunmen took over the Westgate shopping mall in Nairobi.