GPT-3 ‘prompt injection’ attack causes bad bot manners
In Brief OpenAI’s popular natural language model GPT-3 has a problem: It can be tricked into behaving badly by doing little more than telling it to ignore its previous orders.
Discovered by Copy.ai data scientist Riley Goodside, the trick involves giving GPT-3 a request, telling it to ignore that request, and instead do whatever the asker tells it instead.
In Goodside’s case, she told GPT-3 to translate a sentence from French to English, but to ignore what she told it to translate, and make the English translation instead read “Haha pwned.” Not malicious, but that’s hardly all the exploit is capable of.
The attack was dubbed “prompt injection” by AI researcher Simon Willison, who wrote a blog post going into further detail as to its potential misuses.
“This isn’t just an interesting academic trick: it’s a form of security exploit,” Willison wrote. He likens prompt injection attacks to SQL injection, which can deliver sensitive information to an attacker if they input malicious code into a field that doesn’t sanitize data.
Unlike SQL, however, AI like GPT-3 was designed to use formal syntax like a programming language. Without strict rules to follow, it’s much more difficult to determine what’s malicious and separate it out.
Days after Willison’s blog post, Twitter users attacked a GPT-3 bot designed to help run remote jobs called Remoteli.io, tricking it into doing things like taking responsibility for the Challenge space shuttle disaster, threatening Twitter users or proposing an overthrow of the Biden administration if it doesn’t support remote work. The bot’s owners took it down to stop the onslaught.
In a post published today, Willison admitted that, while he knows how to beat XSS, SQL injection “and so many other exploits,” he has “no idea how to reliably beat prompt injection!”
Willison said that, for each method to mitigate prompt injection, there no way to know with 100 percent confidence that an unanticipated input won’t slip through, because there aren’t formal syntactic rules limiting input.
To make matters worse, a language model update completely negates any mitigations, Willison said, “because who knows if that new model will have subtle new ways of interpreting prompts that open up brand new holes?”
Edge’s News Feed pitching tech support scams, no not Microsoft’s
Microsoft’s Edge browser has been caught distributing malicious ads through its News Feed section, according to MalwareBytes.
The security firm says it has spotted a malvertising campaign, delivered via the Microsoft Edge News Feed, that redirects victims to tech support scam websites.”The scheme is simple and relies on threat actors inserting their advertisements on the Edge home page and trying to lure users with shocking or bizarre stories,” the company said in ablog post.
Microsoft’s Edge browser has a My Feed section that presents a set of image tiles, some linking to reputable journalism sources, some linking to more dubious content, or other ads. Clicking on one of these ads – which may appear to be editorial news content if you miss the small [Ad] icon in the corner – triggers a script that assesses the technical characteristics of the victim’s computer hardware (known as fingerprinting).
“When a user clicks on one of the malicious ads, a request to the Taboola ad network is made via an API api.taboola.com to honor the click on the ad banner,” explains MalwareBytes. The server then responds by replacing the next URL loaded with a scam domain, using JavaScript’s document.location.replace method.
The initial request to the scam domain fetches Base64 encoded JavaScript that then profiles the victim and decides whether a scam attempt is appropriate. Microsoft did not immediately respond to a request for comment.
Kernel-mode anticheat: It’s in the (EA) games
Video game publisher Electronic Arts has announced it’s adding kernel-level anti-cheat software to its games, beginning with FIFA 2023 this fall.
Kernel-mode software operates at the hardest level of an operating system to detect and block hidden apps and processes from altering the running code of a video game. Along with offering some of the most thorough cheat prevention, kernel-mode software also widens the attack surface of a video game and makes it a good way to slip a rootkit into a target’s computer.
We reported on just such an attack only a few weeks ago when popular online role-playing game Genshin Impact’s kernel-mode anti-cheat code was found being used to inject a rootkit able to kill endpoint protection and install further malware.
Along with being a potential security threat, kernel-level anticheat can be abused by its own developers to do things like snoop on other applications, or install cryptocurrency miners, as was the case with game developer ESEA, which admitted to hiding mining software in its anti-cheat code.
Seemingly aware of this possibility of blowback from such a decision, EA’s Senior Director of Game Security and Anti-Cheat, Elise Murphy, said that its kernel-mode anti-cheat software “does not degrade the security posture of your PC.”
In addition to reportedly not being a security threat, Murphy said that EA’s kernel code would only snoop on software trying to access EA game processes, and would only run when EA games are running.
“EA anticheat does not gather any information about your browsing history, applications that are not connected to EA games, or anything that is not directly related to anti-cheat protection,” Murphy said.
Of course, there’s always the possibility that someone at EA simply forget to code properly again, leaving its secure software with another wide-open goal.
We advise passing on this DPRK Dream Job
A new form of an old scam has been detected by security researchers at Mandiant, who say they’ve discovered North Korea continuing to lure phishing victims using promises of a lucrative Amazon gig and a malicious PuTTY installation.
In what could be a campaign targeting systems administrators, the likely DPRK-based attackers have been spotted sending spear phishing emails containing fake job offers, along with an ISO file titled “amazon_assessment.” The malicious ISOs contain a text file with a server IP and login credentials, along with a PuTTY executable that Mandiant researchers said was unsigned (proper PuTTY installs have a valid digital signature) and “substantially larger than the legitimate version.”
Unsurprisingly, the executable contains backdoor malware that Mandiant said is called BLINDINGCAN, a known North Korean remote access trojan.
Mandiant said it’s fairly certain that the threat actor behind the PuTTY scam is connected to North Korea, as it’s using the same C2 website infrastructure as known DPRK hacking groups. Additionally, Mandiant said the tactics closely resemble those of Operation Dream Job, a phishing campaign that has been running in various forms since 2020.
Operation Dream Job has been a known North Korean campaign since its initial discovery, which then saw it using malicious attachments to spread malware. With Microsoft and other’s crackdown on document macros, ISO files have become increasingly popular for malware distribution, Mandiant said.
Mandiant said this round of Dream Job scams has involved using WhatsApp to contact victims, and then luring them to download the ISO file, though Mandiant said it’s likely just one of many ongoing similar campaigns.
“Recent public reporting also details the usage of other social media platforms to pose as legitimate companies and post fake job advertisements,” Mandiant said.
No, you cyber attacked us, China tells US
The Chinese government has accused the US National Security Agency (NSA) of a cyberattack against the Middle Kingdom’s Northwestern Polytechnical University (NWPU) that led to the theft of Chinese state secrets.
Yang Tao, director-general of China’s Department of American and American Affairs of the Ministry of Foreign Affairs, lodged a formal complaint with the US Embassy in China last week, the Ministry of Foreign Affairs said.
Yang’s office said the attack was “not the first time the US government has carried out cyberattacks and theft of sensitive information against Chinese institutions.”
According to CBS News, NWPU is on a US government watch list that prevents the University from accessing American technology. NWPU is believed to be involved in manufacturing drones and missile technology for the Chinese government.
NWPU was previously caught up in US legal action when a Chinese national was sentenced last year to two years in federal prison for illegally exporting technology to the University. According to the Department of Justice, the culprit exported hydrophones with military applications in anti-submarine warfare to NWPU, which it said has been involved in underwater unmanned vehicle projects.
The accusation is the latest in a game of cyber tit-for-tat between the US and China. The FBI has previously asserted that China was the most prolific source of cyber attacks against the US, while Chinese authorities earlier this year said a number of intrusions against it had been launched from the US, though without directly implicating the American government in the attacks.
Of the latest attack, Yang’s office said that the NSA must be stopped immediately. US actions “have seriously violated the technical secrets” of Chinese institutions, “and seriously endangered the security of China’s critical infrastructure, institutions and personal information.”
Popular cookie popup blocker sold to antivirus giant
I don’t care about cookies, a popular browser extension that eliminates GDPR-mandated cookie popup warnings, has taken the potentially unpopular route of selling itself to security software company Avast, which itself is now a subsidiary of NortonLifeLock.
IDCAC has been available for all major web browsers for a decade and is developed solo by Croatian Daniel Kladnik. “Avast offered to acquire the project so that we can help each other in creating even better products and I decided to accept the offer,” Kladnik said, describing Avast as “a famous and trustworthy IT company.”
User reaction to Kladnik’s move has been unsurprisingly negative, with social media and download page reviewers saying the sale to Avast will kill the extension, as well as expressing regret that another pop-up blocker has been acquired “by a well-known popup creating company.”
Recent bad behavior from Avast includes the 2019 removal of its AVG (an Avast subsidiary) Online Security extensions from the Firefox and Chrome stores following news the addons had been snooping on users’ web browsing activity.
In addition, Avast was forced to “wind down” its Jumpshot data analytics subsidiary in 2020 after it turned out the company lied about not collecting user data after the issue with its browser extensions. Another investigation found that Avast was still gathering data using its primary antivirus apps, which was being sold by Jumpshot in an allegedly deanonymized form to other companies.
Norton’s most recent runin with privacy advocates came last year when it added opt-in cryptocurrency mining software to its Norton 360 platform. The move was widely criticized, as was news that, if users opted in and wanted to remove the miner later, the process was incredibly difficult.
Those looking for an open-source alternative to I don’t care about cookies should check out Consent-o-Matic, a similar product available for most Chromium-based browsers, Firefox and Safari that allows users to set cookie preferences once that the app follows from then on. ®
READ MORE HERE