It’s 2022 and Australia has its first artificial intelligence scandal.
A freak storm has just hit Melbourne, and an algorithm designed to help emergency services deal with high volumes of requests for help has a stunning and unusual flaw – calls from men are shown to be getting attention faster than calls from women.
It’s an extreme example that would never happen. But when you hear from Catriona Wallace, the chief executive and founder of ASX-listed Flamingo AI, which develops machine learning software for financial services, you start to think it just could be possible.
“Ninety per cent of coders who are actually coding and tagging the data that will go into algorithms are male,” she told The Australian Financial Review’s Innovation Summit this week.
“So already there is a gender bias that may be consciously or not consciously coded into the way the machines operate.”
While politicians rightly consider the big-picture implications of the adoption of machine learning and AI across the economy – particularly the potential for as many as 3.5 million jobs to be affected – those in the industry are considering a host of smaller issues, both practical and philosophical.
The potential for gender bias to be built in AI and automation is one of a broader suite of ethical concerns emerging as the technologies develop and are tested in a host of real world applications that were discussed at the summit.
At CSIRO, a project to get omega-3 out of canola has been underpinned by AI, which helped to identify, isolate, and improve the grains. The agency is also part of a project with 30 water utilities around the world that is using data analytics to predict water pipe failures.
An official from the Australian Securities and Investment Commission even revealed to the summit how AI is being used by the regulator to spot misbehaviour in capital markets. She said the corporate watchdog had been shocked at how fast its algorithm was learning.
But for all the excitement at these developments, there are worries too.
Jobs and Innovation Minister Michaelia Cash and her effective shadow Ed Husic acknowledged the fear in the community about issues such as privacy, and a loss of control.
Concerns about the ethics of AI – which CSIRO is trying to address by helping the government develop an AI code of ethics – go much deeper than machines taking our jobs.
Brock Douglas, an executive with Australian tech group TechnologyOne, who previously worked on IBM’s flagship AI project Watson, says many of the ethical dilemmas come back to transparency.
What is the data that is being used to underpin an AI model? Where it is sourced from, and how is it being manipulated?
What is the basis for the algorithms inside the AI model and how have they been developed?
And perhaps most importantly, how has the machine’s learning been guided?
“Who exactly has trained the AI and what biases have they introduced – not deliberately, we all have our own biases. One of the risks with AI is that trainer’s bias becomes embedded in the algorithm,” Douglas told the summit.
This leads to a further question – raised at the summit by ASIC – as to what safeguards need to be placed on AI systems, such that they can be properly audited.
For example, one of the great fears among regulators in financial services is that an incorrect assumption in an algorithm could lead to a mis-pricing of risk, which could then threaten a specific institution or, in the worst case, a part of the financial system.
Wallace, who designs AI systems for financial services companies and as such works in heavily regulated environments, says differences are emerging between newer “white-box” AI systems and older-style “black-box” systems.
Flamingo’s white-box systems use an unsupervised model – where the algorithm can learn on its own – but are heavily “gated” such that the machines cannot respond in a way that the client had not previously approved. Its systems can also be fully audited.
“You can open it up and see exactly what the decision was,” Wallace says. “Traditional AI is black-box AI, where it will take days, weeks, months to find how a machine has made a decision, if it all you can.”
Professor Toby Walsh, who is the scientia professor of artificial intelligence at the University of New South Wales, argues few of the ethical dilemmas thrown up by AI are new.
“A lot of this is just old-fashioned good values,” he told the summit.
But Walsh, who supports a code of ethics, says that while the dilemmas are the same, the level of oversight on AI is very different. He uses perhaps the most prominent example of AI from a consumer point of view: the autonomous car.
“People developing self-driving cars can test them on the public, knowing they are going to be killing people as they do, without any oversight.
“We don’t let drug companies experiment on the public.”
Brook Douglas argues a code of ethics is unlikely to be enough to regulate AI.
He says while it is industry participants who would have to sign up to the code, it would offer little protection to consumers, who will feel the biggest impact of AI, good and bad.
“I would say that a code of ethics doesn’t go far enough. I think there needs to be some policy response as well from a government basis to start looking at the societal implications.”
‘Unfair advantage’ key
Any policy response will also have to take into account how Australia sees its position in the global AI arms race, which a prominent investor said on Thursday was being led by Google, with China just behind. Larry Marshall says Australia will need to find its “unfair advantage” in AI to stay in this race.
But as Michaelia Cash told the summit, there are bigger questions to be asked before that.
“To help find the right balance, we need to identify what our vision as a nation is,” Cash said.
“Where do we draw the line between regulation and innovative entrepreneurship and economic freedom?”
And that’s not a question any algorithm will help us answer.