Victoria Police has downplayed concerns about alleged gangs of African-Australian youths, but also acknowledges there is a small group of core offenders causing trouble.

"The fact is there is a gang issue here and you are not going to make it go away by pretending it doesn't exist," the prime minister told reporters in Melbourne on Tuesday.

"At some point you have to be fair dinkum and you have to acknowledge that there is a concern, people are concerned about it."

Community leaders have said comments about gangs have led to racism and harassment against African Australians in Melbourne.

Home Affairs Minister Peter Dutton sparked ridicule in January when he said people were scared to go out for dinner in Melbourne due to African gang violence.

Mr Turnbull said he wasn't scared to go out for a meal in the city, but had heard concerns from his political colleagues.

"You have to be honest, there are Sudanese gangs in Melbourne," he said.

"No one is making any reflections about Sudanese migrants. I have spoken about the enormous achievements of Sudanese migrants to Australia, in every respect."

Energy Minister and Victorian MP Josh Frydenberg said he was one of those concerned colleagues.

"We need to take determined action to ensure people are feeling safer on the streets and safer in their homes," he told reporters.

"We are a very successful multicultural nation, people have been welcomed from every corner of the earth.

"At the same time we can't deny there is a problem in some communities with the rise of violence."

Mr Turnbull rejected suggestions his party is stoking racism by commenting on the issue.

In January, Victorian Police Minister Lisa Neville said there was a problem with a small group of repeat offenders.

Victoria Police has been given state funding for a gang task force, and hundreds of youth offenders have been arrested since early 2016.

Premier Daniel Andrews in February acknowledged there was a gang problem.

But on Tuesday he rubbished Mr Turnbull's suggestion people were still scared to eat out.

"I would respectfully say I don't think the prime minister knows what he's talking about - we have the best restaurants in the country and they're full. I don't know that I can offer any further comment than that," he said.

A community advisory body has been set up to ensure the state government is engaged with Sudanese Australian youth.

Victorian Equal Opportunity and Human Rights Commission figures released on Monday show race-related complaints are up by 34 per cent and formally-lodged complaints up 76 per cent year-on-year.

Melbourne-based Anglican bishop Philip Huggins has called for the Victorian Liberal party to scrap its anti-gang political election campaign wording, arguing it makes youths "sound like dangerous wild animals".

Australian Associated Press