Use of hands-free phones should be banned from vehicles as well as hand-held ones, road safety campaigners say.
The charity Brake also called for the penalty for calling or texting behind the wheel to be increased from £100 to somewhere between £500 and £1,000.
Brake said a Freedom of Information request showed more than 500,000 people had points on their licence for using a phone or being otherwise distracted.
The government said it was not looking at banning hands-free kits.
The figures were released to mark the start of National Road Safety Week.
Brake's campaign is being supported by the Association of Chief Police Officers, 10 years after the motoring mobile phone ban was introduced.
The road safety charity wants drivers to turn off their phones, and callers to refuse to speak to someone who is on the road.
Brake pointed to research that suggested 98% of motorists were unable to divide their time without it affecting their driving ability.
Using a mobile phone, eating, drinking and smoking were all shown to increase the risk of a crash, Brake claimed.
Julie Townsend, deputy chief executive of Brake, said: "We're living in an age when being constantly connected is the norm.
"More and more of us have smartphones, and find it hard to switch off, even for a minute.
"While there are enormous benefits to this new technology, it's also posing dangerous temptations to drivers to divert their concentration away from the critical task at hand, often putting our most vulnerable road users in danger.
"Many people who wouldn't dream of drink-driving are succumbing to using their phone and other distractions while driving, oblivious that the effect can be similar and the consequences just as horrific."
Imogen Cauthery, 26, was hit by a driver who was texting on his mobile phone, as well as speeding, when she was nine. She was in a coma for 10 days, and suffers seizures and memory loss.
She says she regularly sees people using their phones while driving.
"I just hate it, I get a really angry feeling, thinking what are you doing? You're ready to ruin another life or kill someone. I was very nearly killed. I want my first life back. My first life without all my problems and I can't get that back," she said.
Peter Rodger from the Institute of Advanced Motorists said a ban would be very difficult for the police to enforce and that a cultural change, which made it morally unacceptable to talk while driving, would be more effective.
"We tend to behave in the way everyone around us expects us to and if we can create an environment in which people don't expect that to happen, that is more likely to control how us, as individuals, actually behave," he said.
The government said it had increased fines for using a phone while driving and was looking at how to improve road safety for younger motorists.
The Department for Transport said road safety was a "top priority".
A spokesman said: "The government is determined that police have the powers they need to tackle any form of dangerous driving, including anyone using a mobile phone at the wheel.
"That is why this year the fixed penalty for this offence was increased to £100 and carries three penalty points.
"Police can stop and arrest any driver if they believe they are not in charge of their vehicle, and this includes if the driver is using a hands-free mobile device.
"There are no plans to change the law around the use of hands-free devices but all penalties are kept under review to ensure they are appropriate."