Resecurity reports a rise in attacks on AI Conversational platforms, targeting chatbots that use NLP and ML to enable automated, human-like interactions with consumers.
Resecurity reported hike in malicious campaigns targeting AI agents and Conversational AI platforms that leverage chatbots to provide automated, human-like interactions for consumers. Conversational AI platforms are designed to facilitate natural interactions between humans and machines using technologies like Natural Language Processing (NLP) and Machine Learning (ML). These platforms enable applications such as chatbots and virtual agents to engage in meaningful conversations, making them valuable tools across various industries.
One of the key categories of Conversational AI platforms is AI-powered Call Center Software and Customer Experience Suites. Such solutions utilize purpose-built chatbots to interact with consumers by processing their input and generating meaningful insights. The implementation of AI-powered solutions like these is especially significant in fintech, e-commerce, and e-government, where the number of end consumers is substantial, and the volume of information to be processed makes manual human interaction nearly impossible or, at least, commercially and practically ineffective. Trained AI models optimize feedback to consumers and assist with further requests, reducing response times and human-intensive procedures that could be addressed by AI.
Resecurity detected a notable interest from both the cybercriminal community and state actors toward conversational AI platforms due to the large number of consumers and the massive volumes of information processed during interactions and personalized sessions supported by AI agents.
On October 8, 2024, Resecurity identified a posting on the Dark Web related to the monetization of stolen data from one of the major AI-powered cloud call center solutions in the Middle East. The threat actor gained unauthorized access to the management dashboard of the platform, which contains over 10,210,800 conversations between consumers and AI agents (bots).
Stolen data could be used to orchestrate advanced fraudulent activities as well as for cybercriminal purposes using AI. Breached communications between AI agents and consumers also revealed personally identifiable information (PII), including national ID documents and other sensitive details provided to address specific requests. The adversary could apply data mining and extraction techniques to acquire records of interest and use them in advanced phishing scenarios and other cyber offensive purposes.
As a result of the compromise, adversaries could access specific customer sessions and steal data and acquire knowledge about the context of interaction with the AI agent, which could later lead to hijacking. This vector may be especially effective in fraudulent and social engineering campaigns when the adversary focuses on acquiring payment information from the victim using some pretext of KYC verification or technical support from a specific financial institution or payment network. Many conversational AI platforms allow users to switch between an AI-assisted operator and a human – the bad actor could intercept the session and control the dialogue further. Exploiting user trust, bad actors could request that victims provide sensitive information or arrange certain actions (for example, confirming an OTP) that could be used in fraudulent schemes. Resecurity forecasts a variety of social engineering schemes that could be orchestrated by abusing and gaining access to trusted conversational AI platforms.
The end victim (consumer) will remain completely unaware if the session is intercepted by an adversary and will continue to interact with the AI agent, believing the session is secure and that the further course of action is legitimate. The adversary may exploit the trust the victim has in the AI platform to obtain sensitive information, which could later be used for payment fraud and identity theft.
Besides the issue of retained PII stored in communications between the AI agent and end users, bad actors were also able to target access tokens, which could be used by enterprises for the implementation of the service with APIs of external services and applications. According to Resecurity, due to the significant penetration of external AI systems into enterprise infrastructure and the processing of massive volumes of data, their implementation without proper risk assessment should be considered an emerging IT supply chain cybersecurity risk. The experts from Resecurity outlined the need for AI trust, risk, and security management (TRiSM), as well as Privacy Impact Assessments (PIAs) to identify and mitigate potential or known impacts that an AI system may have on privacy, as well as increased attention to supply chain cybersecurity. Conversational AI platforms have already become a critical element of the modern IT supply chain for major enterprises and government agencies. Their protection will require a balance between traditional cybersecurity measures relevant to SaaS (Software-as-a-Service) and those specialized and tailored to the specifics of AI, highlighted the threat research team at Resecurity. The EU AI Act and other regulatory frameworks in North America, China, and India are already establishing regulations to manage the risks of AI applications, and enterprises should be aware of these developments, added the cybersecurity company. For example, the recent PDPC AI Guidelines in Singapore already encourage businesses to be more transparent in seeking consent for personal data use through disclosure and notifications. Businesses have to ensure that AI systems are trustworthy, thereby providing consumers with confidence in how their personal data is used.
