AI Jailbreaking Interest Surged 50% in 2024 – How Defenders Can Keep Up
Over the past year, KELA observed cybercriminals constantly sharing and spreading new jailbreaking techniques on underground cybercrime communities. The rise of jailbreak techniques poses a growing threat to AI-driven organizations, enabling users to bypass safety measures, generate harmful content, and access unauthorized data.
Updated March 31, 2025.
What if a cybercriminal could manipulate AI to generate phishing emails, write malicious code, or even help orchestrate an attack – all with just a few simple prompts? This isn’t a distant threat, but instead today’s reality. Cybercriminals use malicious AI tools or try to bypass the guardrails of public GenAI tools at an increasing rate. In fact, the mentions of “jailbreaking” in underground forums surged by 50% in 2024, signaling a growing threat to organizations worldwide.
From crafting deepfake scams to automating malware development, these threats are rapidly escalating. With defenders scrambling to keep up, the real question is: who will win the battle – AI-powered attackers or the security teams trying to stop them? Read on to explore the latest threats, emerging attack techniques, and how organizations can stay ahead in the fight against malicious AI.
Jailbreaking: The Silent Manipulation of AI Applications
Over the past year, KELA observed cybercriminals constantly sharing and spreading new jailbreaking techniques on underground cybercrime communities. AI jailbreaking refers to the process of bypassing safety restrictions programmed into AI systems. Threat actors show their creativity and spread new jailbreaks as new models’ versions are released.
The rise of jailbreak techniques poses a growing threat to AI-driven organizations, enabling users to bypass safety measures, generate harmful content, and access unauthorized data.
Dark AI tools gain popularity in the cybercrime underground
Dark AI tools are jailbroken versions of public GenAI tools like ChatGPT or an open-source LLM without guardrails, which are used by cybercriminals to orchestrate highly targeted cyber attacks. These tools allow threat actors to perform illegal activities, from generating malicious code, creating phishing templates and exploiting new vulnerabilities.
malicious AI tools lower the cost and skill barrier to enter the cybercrime ecosystem. This enables inexperienced individuals to execute cyber attacks by equipping them with advanced hacking tools and malicious scripts for sophisticated campaigns. One of the most popular tools in 2024 was WormGPT, which is an uncensored AI tool that helps cybercriminals launch cyber attacks as business email compromise (BEC) attacks and phishing campaigns.
KELA observed that in 2024 threat actors were constantly looking for WormGPT and other malicious AI tools. There was an increase of 200% in the mentions of malicious AI tools such as WormGPT, WolfGPT, DarkGPT, FraudGPT and others in cybercrime communities compared to 2023.
Exploiting AI in Cybercrime: Key Attack Vectors
Over the past year, KELA observed different tactics used by threat actors using jailbroken public GenAI tools or customized dark AI tools, allowing attackers to prompt the LLM about any topic or task and get malicious content. These are the key attack techniques used by cybercriminals exploiting AI tools:
1. Automated phishing and social engineering
Cybercriminals are increasingly leveraging AI tools to enhance the effectiveness of phishing and social engineering campaigns through text, audio, and images. With AI, attackers can automate the creation of highly convincing fake emails, messages, voice recordings, and even deepfake videos that closely mimic legitimate communications from trusted sources, increasing the likelihood of tricking users into providing their credentials.
Research published by Harvard Business Review showed that AI automated phishing attacks reduce the costs of phishing attacks by more than 95% while achieving equal or greater success rates.
2. Vulnerability research
Another way that cybercriminals used AI to enhance their operations is through automated scanning and analysis. Attackers leverage AI-driven tools to automate penetration testing, uncovering vulnerabilities in security defenses that can be exploited easily. This speeds up the attack cycle and enables cybercriminals to launch an attack before security teams can respond effectively. For example, KELA observed that threat actors utilized the Chinese model, DeepSeek, to exploit software vulnerabilities.
3. Malware and exploit development
Moreover, cybercriminals are increasingly using AI to enhance malware development, making it more adaptable and difficult to detect. Threat actors optimize their payload using AI tools to get assistance with creation of malicious scripting and evasion techniques.
4. Identity Fraud and Financial Crimes:
In December 2024, the FBI warned that cybercriminals are increasingly exploiting generative AI to enhance fraud schemes, making them more convincing and widespread. KELA has also observed that financially motivated actors who promote malicious AI tools focusing on fraud and scam activities. In July 2023, FraudGPT has been promoted on cybercrime forums by the actor “canadiankingpin”, offering AI chatbot with various malicious capabilities, from generating phishing pages to finding VBV BINs, allowing fraudsters to use stolen card details without triggering extra security checks. The tool has been circulated on cybercrime forums in 2024, sparking interest among threat actors.
5. Automated Cyber Attacks:
AI malicious tools allow attackers to automate and execute attacks more easily. Password cracking is one example, machine learning models can be trained on leaked password databases to predict likely passwords or password patterns, vastly speeding up their brute-force attempts. AI tools can also improve credential stuffing (automated use of stolen passwords across many sites). In the realm of network attacks, AI helps optimize Distributed Denial-of-Service (DDoS) strategies.
Fight AI with AI
The rise of AI-powered cyberattacks against organizations demands a shift beyond traditional defenses and visibility into the recent tactics and AI techniques employed by threat actors. Check out KELA’s new report, 2025 AI Threat Report: How cybercriminals are weaponizing AI technology for all the latest findings on AI threats and ways defenders can Fight AI with AI. You can also view our recent webinar on the Dark Tools in AI.
