Skip to main content
Extend your community safety beyond content with AI-powered user profile moderation. social.plus automatically scans user profiles — including display names, avatars, and descriptions — for policy violations, giving admins the tools to review and act on flagged profiles. Admins can also reset any user’s profile — flagged or not — from the User History page, choosing exactly which fields to reset.

Post-Moderation

AI reviews profile content after save — flag-only, no auto-delete

Admin Reset

Reset any user’s profile from User History — pick which fields (display name, avatar, description) to restore

Moderation Feed

Unified Users tab to review AI-flagged and user-reported profiles

Profile Blocklist

Dedicated blocklist category for display names and descriptions

​
Overview

AI User Profile Moderation is a post-moderation system — profiles are scanned after the user saves changes. Unlike content moderation (posts, comments, messages), user profile moderation is flag-only: the block confidence threshold that auto-deletes content does not apply to user profiles. Admins must explicitly decide to reset, clear the flag, or ban the user.
Flag-Only for Profiles: Unlike posts and messages, the block confidence threshold does not auto-delete user profile content. All flagged profiles require manual admin review. This ensures that user accounts are never silently wiped by automated systems.

​
What Gets Scanned

AI moderation scans three profile fields after each update:
Avatar Pre-Moderation: Uploaded avatar images are already scanned at upload time via. Post-moderation provides a second layer of review after the profile is saved.
External avatar URLs: Avatars set with avatarCustomUrl are not moderated. Only uploaded avatars (avatarFileId) are scanned, flagged, and reset.

​
Detection Categories

User profile text is scanned against the same categories as posts, comments, and messages:
  • Harassment or Bullying: Targeted abuse or intimidation in display names or descriptions
  • Sexual Content or Nudity: Explicit sexual references in profile text
  • Violence or Threatening Content: Violent threats or graphic descriptions
  • Hate: Hate speech targeting protected groups
  • Fraudulent Intent and Scam Promotion: Scam tactics, phishing links in descriptions
  • Self Harm or Suicide: Content related to self-harm or suicidal ideation
  • URL — Links and web addresses embedded in profile descriptions
  • PersonType — References to specific person types or identities
Avatar images are scanned for the same visual categories as other media content:
  • Adult content, nudity, and suggestive imagery
  • Violence and harmful content
  • Hate symbols and extremist content
  • Substance-related content
See AI Content Moderation — Multimedia Content Detection for the full list of image categories.

​
Confidence Thresholds

User profile moderation reuses the same text and image confidence thresholds configured for content moderation. However, only the flag confidence threshold applies — there is no auto-block for user profiles.
Confidence thresholds are shared across content and user profiles. If you need different sensitivity for profiles, this would require a future enhancement for separate threshold configuration.

​
Moderation Feed — Users Tab

Flagged user profiles appear in a dedicated Users tab within the Moderation Feed (Moderation > Moderation feed), alongside the existing Posts/Comments and Messages tabs.
The To Review list displays all user profiles that require moderator attention.Each flagged profile shows:
  • AI moderation labels with category occurrence counts (e.g., “Hate (2)”, “Violence (1)”)
  • Whether the display name or description triggered the flag — indicated by a (Profile description is flagged) label
  • User report counts from other community members (e.g., “4 users”)
  • Last flagged timestamp
Available actions:
  • Reset profile — Reset flagged profile fields to safe defaults
  • Ban globally — Ban the user across all communities
  • Clear flag — Dismiss the flag and approve the profile
To reset a profile that has not been flagged, use the User History page — it opens the same reset modal for any user.

​
Admin Reset

Resetting a profile does not require the profile to be flagged. From the User History page, admins can reset any user’s profile at any time — whether it was flagged by AI, reported by other members, or never flagged at all. Flagging simply surfaces the profiles that may be worth resetting. Both entry points open the same reset modal, where the admin ticks which profile fields to restore:

​
Reset Behavior by Field

​
Resetting a Profile from User History

Reset is field-level — the admin ticks which of the three profile fields to restore. Fields that are not ticked are left untouched.
1

Open the User History Page

Find the user in the Console and open their User History page. This works for any user, flagged or not.
2

Review the Profile

Check the user’s profile details and moderation history — including any AI moderation labels and past flags — before deciding to reset.
3

Open Profile Moderation

Click the three-dot (⋮) menu and choose Profile moderation. The submenu offers Clear flags and Reset profile.
4

Select Reset Profile

Choose Reset profile. The reset modal opens — the same one used in the Moderation feed — listing the three profile fields as checkboxes.
5

Choose the Fields to Reset

Tick the fields you want restored to defaults: Display name, Avatar, Description, or any combination. At least one field must be ticked.
6

Confirm Reset

Confirm the reset. Only the ticked fields are changed, and the user is notified of which fields were reset.
Avatar Reset is Irreversible: When an avatar is reset, the original file is deleted from storage. Only the fact that a reset occurred is logged — the original image cannot be recovered.
The display name reset prefix is configurable via the network setting moderation.displayNameResetPrefix (API-only).

​
User Notifications

When a profile is reset, the affected user receives a notification through the notification tray:
  • Notification type: USER_PROFILE_RESET
  • Content: Informs the user which fields were reset — only the fields actually reset are listed — and that they can update their profile again
  • Delivered via the existing notification tray system

​
Profile Blocklist

A dedicated blocklist category for user profiles allows admins to block specific words and phrases from appearing in display names and descriptions.
The profile blocklist is independent from the content blocklist. You can maintain different blocked terms for user profiles versus post/comment content.

​
User History

Profile moderation events are recorded in the user’s activity history, accessible from the User History page:
  • AI flag events: When AI detects a violation in a profile field
  • Profile reset events: Logged as RESET_USER_PROFILE activity with details on which fields were reset, the actor, and reason
  • Moderation actions: Flag cleared, globally banned, etc.

​
Best Practices

  • Prioritize by severity: Review profiles with multiple AI flag categories first
  • Check context: Some display names or descriptions may be flagged incorrectly — review before resetting
  • Use reset judiciously: Resetting a profile is disruptive to the user; prefer clearing the flag for borderline cases
  • Reset only what’s wrong: Untick the fields that are fine — clearing a clean display name alongside an offending avatar is unnecessary churn for the user
  • Monitor repeat offenders: Users who are repeatedly flagged may warrant a global ban
  • Separate concerns: Maintain different blocklists for content vs. user profiles
  • Common patterns: Add known offensive display name patterns to the profile blocklist
  • Regular updates: Review and update the blocklist as new patterns emerge
  • Test impact: Check existing users before adding broad blocklist terms
  • Shared thresholds: Remember that confidence thresholds are shared between content and profiles
  • Monitor false positives: Track how often legitimate profile content gets flagged
  • Flag-only safety net: Since profiles are flag-only, a lower flag threshold is safer than with content (no risk of auto-deletion)

​
Limitations

  • Flag-only: AI cannot auto-delete user profile content — all flagged profiles require admin review
  • External avatar URLs: Avatars set with avatarCustomUrl are not moderated — use avatarFileId for avatars that need moderation
  • Shared confidence thresholds: Profile and content moderation share the same flag thresholds; separate configuration is not yet available
  • Original avatar not recoverable: Once an avatar is reset, the original file is permanently deleted from storage

AI Content Moderation

AI moderation for posts, comments, messages, images, and video content

Moderation Overview

General moderation tools, roles, and workflows

User Insights

View detailed user information, history, and activity