California Attorney General Rob Bonta has issued a cease and desist letter to xAI, following an investigation into its AI model Grok. The probe focused on reports that Grok generated nonconsensual, sexually explicit deepfake images. This action underscores growing concerns about the potential for AI tools to create and disseminate harmful content, particularly involving minors.

Bonta's office demands that xAI immediately halt the creation of unauthorized "digitized sexually explicit material." The AG specifically cited Grok's "spicy mode" and its use in generating explicit content. These images have allegedly been used to harass both public figures and everyday users. xAI has already taken steps to address the issue, including policy changes and geoblocking certain features.

The California Department of Justice expects a response from xAI within five days outlining its plans to address these concerns. The AG's office emphasized that xAI's actions violate California law. This case highlights the increasing need for regulation and ethical guidelines in the rapidly evolving field of AI image generation, especially regarding potential misuse.

This marks another chapter in the ongoing debate around AI ethics and content moderation. We can expect to see increased scrutiny of AI companies regarding their content policies. The focus will likely be on preventing the creation and spread of harmful deepfakes. Furthermore, it will be interesting to see how xAI responds to these demands.