Generative AI is changing cybercrime less by creating entirely new motives than by making old tactics faster, cheaper and easier to run across borders. Researchers cited in the source describe a criminal market that prizes convenience: if a tool improves output and lowers friction, it is likely to spread.
Vincenzo Ciancaglini, a senior threat researcher at Trend Micro, says malicious actors are using generative AI as a productivity tool. Mislav Balunović, an AI security researcher at ETH Zurich, points to phishing and doxxing as areas where language models can be especially useful to attackers.
Criminals are choosing tools that work now
The source describes a shift away from criminals building their own AI systems. Last year saw the rise and fall of WormGPT, an AI language model built on top of an open-source model and trained on malware-related data. It was designed to help hackers and operated without ethical rules or restrictions.
But last summer, WormGPT’s creators announced they were shutting it down after it attracted media attention. Since then, the source says cybercriminals have mostly moved away from developing their own AI models and toward tricks with existing tools.
The reason is practical. Ciancaglini says criminals want easy gains and will only adopt new technology if it offers better rewards than what they already use. If a new tool raises the risk of getting caught without clearly improving results, it is less attractive.
Phishing is the biggest current use case
The biggest criminal use case for generative AI right now is phishing, according to Balunović. Phishing attempts to trick people into revealing sensitive information that can then be used for malicious purposes.
The source says researchers have found that the rise of ChatGPT has been accompanied by a huge spike in the number of phishing emails. Spam-generating services such as GoMail Pro have integrated ChatGPT, letting criminal users translate or improve messages sent to victims.
That matters because poor language once helped people spot scams. Ciancaglini says older Nigerian prince scams were often easier to identify because the English was clumsy and full of grammatical errors. Language models now let scammers create messages that sound as if they were written by a native speaker.
Better AI translation can also help criminal groups communicate with each other across countries. The risk, according to Ciancaglini, is that groups could coordinate large-scale operations that move beyond their own nations and target victims elsewhere.
OpenAI says its policies restrict people from using its products for illegal activities. The company says it uses human reviewers and automated systems to identify misuse, and can issue warnings, temporary suspensions and bans. In a report from February, OpenAI said it had closed five accounts associated with state-affiliated actors.
Deepfakes are entering the scam economy
Generative AI has also made synthetic images, videos and audio more realistic. Criminals have noticed, and the source says deepfakes are now being marketed in underground spaces.
Earlier this year, an employee in Hong Kong was reportedly scammed out of $25 million after cybercriminals used a deepfake of the company’s chief financial officer to persuade the employee to transfer money to a scammer’s account.
Ciancaglini’s team found people on platforms such as Telegram showing portfolios of deepfakes and selling services for as little as $10 per image or $500 per minute of video. One of the most popular people for criminals to deepfake is Elon Musk, according to Ciancaglini.
Audio deepfakes are a particular concern because they are cheap to make and need only a couple of seconds of someone’s voice, which could be taken from social media. The source says there have been high-profile cases in the US where people received distressing calls that appeared to come from loved ones claiming to have been kidnapped and asking for money, only for the caller to be a scammer using a deepfake voice recording.
Ciancaglini recommends that loved ones agree on a regularly changing secret safe word to help confirm identity during alarming calls. The point is simple: when a voice can be imitated, trust needs another check.
Identity checks and jailbreak services are new markets
Deepfakes are also being used to bypass know your customer verification systems. Banks and cryptocurrency exchanges use these systems to confirm that customers are real people, often by asking new users to take a photo of themselves holding a physical identification document in front of a camera.
The source says criminals have started selling apps on Telegram that can get around the requirement. These apps offer a fake or stolen ID and impose a deepfake image on top of a real person’s face to fool the verification system on an Android phone’s camera. Ciancaglini found examples of people offering these services for cryptocurrency website Binance for as little as $70.
For now, Ciancaglini says the techniques are still fairly basic and similar to Instagram filters, where someone else’s face is swapped for your own. He expects criminals to move toward actual deepfakes that could support more complex authentication.
Another emerging market is jailbreak-as-a-service. AI companies have built safeguards to stop models from producing harmful or dangerous information. Jailbreaking tries to manipulate those systems into generating outputs that violate policy, such as ransomware code or scam-email text.
Services such as EscapeGPT and BlackhatGPT offer anonymized access to language-model APIs and frequently updated jailbreaking prompts. AI companies such as OpenAI and Google then have to keep plugging security holes, while malicious actors keep testing new prompts.
Doxxing risk grows with online data
Language models can also help with doxxing, which means revealing private identifying information about someone online. Balunović says these systems are trained on vast amounts of internet data, including personal data, and can infer details from small clues.
A chatbot could be prompted to act like a private investigator and analyze text written by a target. From ordinary details, it might infer age from when someone went to high school or location from landmarks mentioned on a commute. The more information available online, the more exposed a person can become.
Balunović was part of a team of researchers that found late last year that large language models such as GPT-4, Llama 2 and Claude can infer sensitive information such as ethnicity, location and occupation from mundane chatbot conversations. Since their paper came out, the source says new services exploiting this capability have emerged.
The existence of those services does not itself prove criminal activity. But it shows the capability is available. Balunović says defenses are the way to prevent abuse, and that companies should invest in data protection and security. For individuals, the practical lesson is to think carefully about what personal details they share online.