Separating parents in England and Wales could face fines for refusing to try mediation under new proposals.
The government is set to make sessions mandatory and offer funding support in an effort to divert cases away from under-pressure family courts.
Justice Secretary Dominic Raab said "lengthy and combative courtroom battles" were harmful for children.
But the Law Society said the plan could put victims of undetected coercive control in a vulnerable position.
The government has launched a consultation on the policy, which it hopes will help to deal with backlogs in the family courts system.
The policy will apply to separations where children are involved but the government is also looking more broadly at compulsory mediation for those without children, and how that could be funded.
Mediation, where couples can agree the practicalities of a split in the presence of an independent specialist, is already available as an option, but there is no requirement to engage with it prior to going to court.
Under the plans, judges will be able to order separating couples to make a "reasonable attempt" to agree matters such as child custody and financial arrangements in mediation, and will face fines "if they act unreasonably and harm a child's wellbeing by prolonging court proceedings".
The new rules, which apply to couples with children who are married or in civil partnerships, will exclude relationships where there has been domestic abuse.
But the Law Society, which represents solicitors in England and Wales, has warned the policy "risks sending people to mediation who should not be there".
Richard Miller, the organisation's head of justice, told the BBC mediation was "absolutely vital" in many cases, but in situations where there was an "imbalance of power", it could lead to "unjust outcomes".
He added: "Relationships where there is manipulation or coercive control can be very difficult to spot initially, and those on the receiving end of that sort of abuse might not even understand the extent to which they are being controlled themselves."
Dominic Raab said the plans would save the courts time and protect children
Women's Aid has said clarity is "urgently needed" to understand how the Ministry of Justice will ensure all domestic abuse survivors will be kept safe and allegations will be properly investigated.
The charity's head of policy, Lucy Hadley, said survivors "frequently fear they will be accused of parental alienation" if they raise domestic abuse.
"This lack of disclosure means the proposed exclusion of domestic abuse cases from mediation will not always work, and we fear that women will be re-traumatised by mediation with their perpetrators.
"If mediators don't have a thorough understanding of domestic abuse - or even know abuse is a factor in a case - these processes will ignore unequal power dynamics, exacerbating the abuse women experience and putting them at further risk", Ms Hadley added.
Mr Miller called on the government to reintroduce aid for early legal advice - which was cut from private family cases in 2012. That would make it easier for people to see a solicitor before any mediation in order to provide an extra safeguarding measure, he said.
It is common practice for an assessment to be held prior to mediation in order to determine suitability, and some couples may be referred to the court at that stage.
The consultation on the proposals is also expected to look at whether divorce should be funded under legal aid thresholds.
The Ministry of Justice believes mandatory mediation could help up to 19,000 families a year agree terms without going into the court system, which is dealing with huge backlogs made worse by the Covid-19 pandemic.
According to the most recent data, almost 29,000 new divorce applications were made in the three months between July and September 2022, with demand increasing after the introduction of no-fault divorces last year.
Mr Raab, who recently backed higher sentences for domestic killers who have subjected their victims to coercive control, said it would free up judges to "ensure the most urgent cases involving domestic abuse survivors are heard by a court as quickly as possible."
The announcement was welcomed by the Family Mediation Council (FMC), which said it would help families avoid "the stress and delays of going to court".
Under the plans, sessions with a qualified mediator would be part-funded by the government with a voucher worth up to £500.
According to the FMC, its registered mediators charge an average of £140 per person per hour, but rates and the number of sessions needed vary.
A mediation funding scheme has already supported 15,300 families, the MoJ said, and will be expanded with an extra £15m of government money.
More than two thirds of the first 7,200 users to be supported by the scheme "reached whole or partial agreements away from court", according to the department.
A 12-week consultation on the plans began on Thursday and will close on 15 June.