By Barbara Ortutay
Roblox, an online gaming platform that is extremely popular with children and teenagers, is rolling out an open source version of an artificial intelligence system that says it will help preemptively detect plunder languages in gaming chat.
The move accused the company of not enough to protect children from predators as it faces lawsuits and criticism. For example, a lawsuit filed in Iowa last month claims that a 13-year-old girl was introduced to an adult Roblox predator and was subsequently invited and raped in multiple states. The lawsuit, filed in Iowa District Court in Polk County, alleges that Roblox’s design features have become children who use “pedophiles’ simple prey.”
By default, Roblox strives to make the system as safe as possible, but says, “systems are not perfect, and one of the industry’s biggest challenges is detecting serious harms, such as potential child dangers.”
An AI system called Sentinel helps detect early signs of possible child danger, such as sexually exploitative language. Roblox said the system began filing 1,200 reports in the first half of 2025 of potential child exploitation and National Centers for Exploited Children. The company is currently in the open sourcing process so that other platforms can use it as well.
Preemptive detection of possible risks to children requires caution for AI systems and humans as the conversation may appear harmless at first. Questions like “How old are you?” or “Where did you come from?” You don’t necessarily raise the red flag yourself, but if you put it in the context over a longer course of conversation, they can take on a different meaning.
With over 111 million users each month, Roblox doesn’t allow users to share videos and images in chat, and like most moderation rules, it cannot attempt to block personal information such as phone numbers.
Additionally, children under the age of 13 may not chat with other users outside of the game unless they have express parental permission. Unlike many other platforms, private chat conversations are not encrypted, so you can monitor and moderate them.
“We’ve been putting filters in place, and these filters tend to focus on what’s said within a single text or a few lines of text. And it’s really good for blocking blasphemy and blocking different types of abusive language and things like that.” “But when you’re thinking about things related to child danger and grooming, the type of behavior you’re looking at appears for a very long time.”
Sentinel captures snapshots of a one-minute chat across around 6 billion messages per day, analyzing potential harm. To do this, Roblox says it has developed two indexes. One consists of benign messages and chats determined to include risk violations for children. Roblox says this allows the system to recognize harmful patterns beyond simply flagging specific words or phrases, but also making the entire conversation a context.
“That index gets better as you detect worse actors. You keep updating that index on a continuous basis. Then is there another example of what a regular user does?” said Naren Koneru, Vice President of Engineering for Trust and Safety at Roblox.
When users chat, the system maintains a score. Is it closer to a positive or negative cluster?
“It just sends one message, so it doesn’t happen with one message, but it happens because every day interaction leads to one of these two,” Koneru says. “Then we say, okay, this user may be someone we need to see more. And we bring out all their other conversations, the other friends, the games they’ve played, and all of them.”
Humans will then consider the flag to dangerous interactions and law enforcement.
Original issue: August 7, 2025, 5:34pm EDT
