A deep dive into the engineering challenges of making multiple AI models aware of each other in collaborative chat environments.
Introduction
Most AI chat applications assume a simple interaction model: one user, one AI. But what happens when you want Claude and GPT-4 to participate in the same conversation? Or when a team of humans and multiple AI assistants collaborate in real-time?
At CoChat, we built exactly this. Our platform enables group chats where multiple AI models coexist alongside human participants. This sounds straightforward until you realize that each AI model has no inherent awareness that other AIs exist in the conversation—let alone which responses came from which model.
This post dives deep into how we solved this problem using a filter chain architecture that transforms every request to provide contextual awareness to each AI model about who’s in the conversation and who said what.
The Challenge: Why Multi-AI Awareness is Hard
The Fundamental Problem
When an AI receives a conversation history, it looks something like this:
system: You are a helpful assistant.
user: What's the capital of France?
assistant: Paris is the capital of France.
user: Thanks! Can you tell me more about it?
assistant: Paris is known for the Eiffel Tower...From the AI’s perspective, all assistant messages came from itself. It has no way of knowing that the first response came from Claude Sonnet while the second came from GPT-4o. This creates several problems:
- Attribution confusion: When a user asks “What did you say earlier about Paris?”, the AI assumes it made that statement, even if another model did.
- Capability mismatches: If GPT-4o referenced a tool only it has access to, Claude might hallucinate that it has the same capability.
- Contradictory statements: AIs might contradict each other without acknowledging the disagreement, confusing users.
- Context window limits: Long multi-model conversations fill context windows faster than anticipated.
The Constraints
Our solution needed to satisfy several competing requirements:
- Model agnostic: Work with any LLM backend (OpenAI, Anthropic, Ollama, etc.)
- Non-invasive: Don’t require special API features or model fine-tuning
- Efficient: Minimize added tokens and latency
- Maintainable: Easy to extend as we add features
- Fail-safe: Degrade gracefully if components fail
Architecture Overview: The Filter Chain Pattern
CoChat was built on top of Open WebUI which makes use of the filter chain architecture—a pattern borrowed from servlet programming that processes requests through a series of modular transformations.
The Filter Chain Concept
Each filter is a Python class with inlet (pre-processing) and optionally outlet (post-processing) methods:
class Filter:
class Valves(BaseModel):
priority: int = Field(default=0)
enabled: bool = Field(default=True)
async def inlet(self, body: dict, **kwargs) -> dict:
# Transform request before it reaches the LLM
return body
async def outlet(self, body: dict, **kwargs) -> dict:
# Transform response after LLM generates it
return bodyFilters are sorted by priority (lower runs first) and executed sequentially:
Request → Filter(-100) → Filter(-60) → Filter(-50) → Filter(50) → LLM → ResponseOur Multi-AI Awareness Filter Chain
┌─────────────────────────────────────────────────────────────────┐
│ Incoming Request │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ base_system_prompt_filter (priority: -100) │
│ ─────────────────────────────────────────── │
│ • Injects global system prompt with multi-model awareness │
│ • Explains the [modelname] convention to the AI │
│ • 100% static content for prompt caching │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ message_attribution_filter (priority: -60) │
│ ─────────────────────────────────────────── │
│ • Adds [Username at TIME]: prefix to user messages │
│ • Adds [ModelName] prefix to assistant messages │
│ • Looks up model names from database │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ collaborative_context_filter (priority: -50) │
│ ─────────────────────────────────────────── │
│ • Injects participant roster for group chats │
│ • Includes user bios and roles │
│ • Adds per-user tool availability context │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ conversation_context_filter (priority: 50) │
│ ─────────────────────────────────────────── │
│ • Monitors token usage against context limits │
│ • Compresses old messages into summaries when needed │
│ • Injects search tool for retrieving original messages │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ LLM Backend │
└─────────────────────────────────────────────────────────────────┘Implementation Deep Dive
1. Base System Prompt: Teaching AIs About Multi-Model Conversations
The foundation is a carefully crafted system prompt that explains the multi-model context to every AI. This runs first (priority -100) and remains completely static to enable prompt caching.
Key section from base_system_prompt_filter.py:
DEFAULT_BASE_PROMPT = """You are a helpful AI assistant, created by CoChat Inc.
## Context Awareness
Session context (model, user preferences) is provided in a [Context] message
at the start of each conversation. Message timestamps are shown as
[Name at TIME]: format. Use this context to personalize your responses.
...
### Multi-Model Context Awareness
In some conversations, you may see assistant messages from different AI models.
These are indicated by `[modelname]` prefixes in the message content (without
a colon, unlike user messages which show `[username]:`).
When you see assistant messages with model name prefixes:
- Do not assume those messages were written by you
- Each model may have different capabilities, knowledge cutoffs, and
reasoning patterns
- If asked about a previous assistant response, acknowledge it came from a
different model if the [modelname] prefix differs from the model identified
in your session context
- You can reference, build upon, or correct information from other models'
responses as appropriate
The `[modelname]` prefix is for your context only - users do not see these
prefixes in their interface.
"""Why static prompts matter:
Many AI APIs support prompt caching—if the system prompt is identical across requests, providers can cache the processed prompt and reduce latency/cost. By keeping the base prompt 100% static (no user names, timestamps, or dynamic content), we enable this optimization:
class Filter:
class Valves(BaseModel):
priority: int = Field(default=-100)
base_prompt: str = Field(
default=DEFAULT_BASE_PROMPT,
description="Base system prompt. 100% static for prompt caching.",
)
async def inlet(self, body: dict, **metadata) -> dict:
# Skip background tasks - they have specialized prompts
if __metadata__ and __metadata__.get("task"):
return body
messages = body.get("messages", [])
# Deduplication: check if already injected
if messages and messages[0].get("role") == "system":
existing = messages[0].get("content", "")
fingerprint = self.valves.base_prompt[:100]
if fingerprint in existing:
return body # Already present
# Inject as system message
if messages and messages[0].get("role") == "system":
# Prepend to existing system prompt
messages[0]["content"] = f"{self.valves.base_prompt}\n\n{existing}"
else:
# Insert new system message
messages.insert(0, {"role": "system", "content": self.valves.base_prompt})
return body2. Message Attribution: The [ModelName] Convention
The attribution filter transforms the raw conversation history to tag each message with its source. This is where the magic happens for multi-model awareness.
The transformation:
Before:
───────
user: What is quantum computing?
assistant: Quantum computing uses quantum mechanics...
user: Can you explain qubits?
assistant: A qubit is the fundamental unit...
After (what the AI sees):
──────────────────────────
user: [Alice at 2:30 PM]: What is quantum computing?
assistant: [Claude Sonnet 4] Quantum computing uses quantum mechanics...
user: [Alice at 2:31 PM]: Can you explain qubits?
assistant: [GPT-4o] A qubit is the fundamental unit...Key implementation details from message_attribution_filter.py:
def _transform_messages_with_attribution(
self, messages: list, db_messages: list, models: dict
) -> int:
"""
Transform messages to add author/model attribution.
Args:
messages: The messages array from the request body
db_messages: Ordered messages from database (for model name lookup)
models: Cache of model ID -> model info mappings
"""
transformed_count = 0
db_idx = 0
for idx, message in enumerate(messages):
role = message.get("role")
# Skip system messages - not in db_messages
if role == "system":
continue
if role == "user" and self.valves.prefix_user_messages:
if message.get("author"):
author_name = message["author"].get("name", "Unknown")
# Get timestamp
timestamp_str = ""
if self.valves.include_timestamps:
timestamp = message["author"].get("timestamp")
if not timestamp and db_idx < len(db_messages):
timestamp = db_messages[db_idx].get("timestamp")
if timestamp:
timestamp_str = self._format_timestamp(timestamp)
# Build prefix: [Alice at 2:30 PM]:
prefix = f"[{author_name} at {timestamp_str}]:" if timestamp_str \
else f"[{author_name}]:"
content = message.get("content", "")
if isinstance(content, str) and not re.match(r"^\[.+?\]:\s*", content):
message["content"] = f"{prefix} {content}"
transformed_count += 1
elif role == "assistant" and self.valves.prefix_assistant_messages:
model_name = None
# Look up model name from database
if db_idx < len(db_messages):
db_msg = db_messages[db_idx]
if db_msg.get("role") == "assistant":
# Use stored modelName directly
if db_msg.get("modelName"):
model_name = db_msg["modelName"]
# Fallback: look up model ID in cache
elif db_msg.get("model"):
model_id = db_msg["model"]
model_name = models.get(model_id, {}).get("name", model_id)
if model_name:
content = message.get("content", "")
if isinstance(content, str) and not content.startswith(f"[{model_name}]"):
message["content"] = f"[{model_name}] {content}"
transformed_count += 1
# Track position in db_messages
if role in ("user", "assistant"):
db_idx += 1
return transformed_countWhy we use brackets, not role metadata:
We considered using name or role metadata fields, but:
- Compatibility: Not all providers support custom role names
- Visibility: In-content prefixes are visible to all models regardless of API quirks
- Simplicity: No special API handling needed—it’s just text
The [ModelName] format (no colon) vs [Username]: (with colon) convention makes it easy for AIs to distinguish AI responses from user messages in the history.
3. Collaborative Context: The Participant Roster
For group chats, knowing who’s participating is crucial. The collaborative context filter injects a roster of participants with their roles and capabilities.
Example injected context:
## Chat Context (Multi-User Mode)
This is a multi-participant conversation. Multiple people are chatting together.
### Active Participants:
- Alice (owner): Product lead, focusing on user experience
- Bob (can write): Backend engineer
- Carol (can read): Design reviewer
## Tool Availability by Participant:
- Alice: calendar, email
- Bob: github, jenkins
- Carol: (no tools)
## Current Request
- Message sender: Bob
- Tools available for THIS request: github, jenkins
## CRITICAL: Tool Usage Rules
- Tools are scoped to the message sender, NOT the conversation
- You can ONLY use tools listed as available for the current request
- Previous tool responses in chat history were made when THEIR owners sent messages
- If asked to use tools you don't have access to: clearly explain you cannot access
those tools for this user
- NEVER fabricate or simulate tool responses - if tools are unavailable, say soThe implementation:
def _build_collaborative_context(
self, chat, owner, current_user, __request__=None, location: str = None
) -> str:
"""Build collaborative chat context with participant roster and tool availability."""
from open_webui.models.users import Users
from open_webui.utils.access_control import get_users_with_access
# Get access control from project or chat level
effective_access_control = chat.access_control
if chat.folder_id:
folder = Folders.get_folder_by_id(chat.folder_id)
if folder and folder.is_collaborative and folder.access_control:
effective_access_control = folder.access_control
# Get participants
if effective_access_control:
read_participants = get_users_with_access(type="read", access_control=effective_access_control)
write_participants = get_users_with_access(type="write", access_control=effective_access_control)
else:
read_participants = []
write_participants = []
# Build participant list with roles and bios
participant_list = []
all_participant_ids = set()
# Add owner first
if owner and can_user_access_chat(chat, owner.id, "read"):
entry = f"- {owner.name} (owner)"
if owner.bio:
entry += f": {owner.bio}"
participant_list.append(entry)
all_participant_ids.add(owner.id)
# Add write participants
for participant in write_participants:
if participant.id not in all_participant_ids:
entry = f"- {participant.name} (can write)"
if participant.bio:
entry += f": {participant.bio}"
participant_list.append(entry)
all_participant_ids.add(participant.id)
# Add read-only participants
for participant in read_participants:
if participant.id not in all_participant_ids:
entry = f"- {participant.name} (can read)"
if participant.bio:
entry += f": {participant.bio}"
participant_list.append(entry)
all_participant_ids.add(participant.id)
# Build tool availability section
# ... (per-user tool enumeration)
return f"""## Chat Context (Multi-User Mode)
This is a multi-participant conversation.
### Active Participants:
{chr(10).join(participant_list)}
{tool_availability_section}
### Message Format
- Messages are formatted as [Name]: content
- Address participants by name when appropriate
- Be aware that different messages come from different people"""Why tool scoping matters:
In collaborative chats, different users have different tool integrations. When Alice asks the AI to “check my calendar,” the AI needs to know that it’s using Alice’s calendar tool—not Bob’s. And if Bob asks the same question, the AI should understand it now has access to Bob’s tools (or lack thereof).
This prevents a dangerous class of bugs where an AI hallucinates tool access because it saw a tool response earlier in the conversation from a different user.
4. Long Conversation Handling: Hierarchical Summarization
Multi-model AI group chats tend to be longer than single-AI chats. Users experiment, compare responses, and the conversation grows. The context filter handles this with a multi-tier approach:
Threshold-based triggers:
| Threshold | Action |
|---|---|
| 60% | Queue background summarization |
| 70% | Queue background embedding |
| 80% | Active compression (summaries replace old messages) |
The compression architecture:
┌─────────────────────────────────────────────────────────────────┐
│ Original Conversation │
│ msg 1, msg 2, msg 3, ... msg 47, msg 48, msg 49, msg 50 │
└─────────────────────────────────────────────────────────────────┘
│
▼ (80% threshold crossed)
┌─────────────────────────────────────────────────────────────────┐
│ Compressed Context │
│ ┌─────────────┐ ┌─────────────┐ ┌────────────────────────┐ │
│ │ Summary of │ │ Summary of │ │ Recent messages │ │
│ │ msgs 1-20 │ │ msgs 21-40 │ │ (msgs 41-50 verbatim) │ │
│ └─────────────┘ └─────────────┘ └────────────────────────┘ │
│ │
│ + search_conversation_history tool for retrieving details │
└─────────────────────────────────────────────────────────────────┘Sticky compression flag:
Once compression is triggered for a chat, it stays enabled even if the conversation temporarily shrinks:
# Check if compression is already enabled (sticky flag)
compression_enabled = Chats.is_compression_enabled(chat_id)
should_compress = should_trigger_compression(
current_tokens, context_limit, self.valves.compression_threshold
)
if not should_compress and not compression_enabled:
# No compression needed - pass through unchanged
return body
# Set sticky flag if not already set
if not compression_enabled:
Chats.set_compression_enabled(chat_id, True)This prevents thrashing where compression repeatedly enables/disables as the conversation hovers around the threshold.
Background task queueing:
Summarization and embedding run as background tasks to avoid blocking the user’s request:
async def outlet(self, body: dict, **kwargs) -> dict:
"""Post-LLM processing: check if we need to queue background work."""
# Get full conversation from DB
chat = Chats.get_chat_by_id(chat_id)
all_messages = self._get_messages_from_chat(chat)
total_tokens = count_messages_tokens(all_messages)
# Queue summarization at 60%
if should_trigger_background_summarization(total_tokens, context_limit, 0.60):
if chat_id not in self._pending_summarization:
self._queue_background_task(
self._background_summarize(chat_id, all_messages, context_limit)
)
# Queue embedding at 70%
if should_trigger_background_embedding(total_tokens, context_limit, 0.70):
if chat_id not in self._pending_embedding:
self._queue_background_task(
self._background_embed(chat_id, all_messages, context_limit)
)
return bodyThe Complete Flow: Request Lifecycle
Here’s what happens when Alice sends a message in a group chat with Bob, using Claude Sonnet:
1. REQUEST ARRIVES
─────────────────
POST /api/chat/completions
{
"model": "claude-sonnet-4-20250514",
"messages": [
{"role": "user", "content": "What about async/await?"},
{"role": "assistant", "content": "Async/await is a pattern..."},
{"role": "user", "content": "Thanks! Can you show an example?"}
],
"chat_id": "abc123"
}
2. BASE SYSTEM PROMPT FILTER (priority: -100)
───────────────────────────────────────────
Prepends static system prompt explaining multi-model awareness.
messages[0] = {
"role": "system",
"content": "You are a helpful AI assistant... [multi-model instructions]"
}
3. MESSAGE ATTRIBUTION FILTER (priority: -60)
────────────────────────────────────────────
Looks up db_messages to find model names and timestamps.
messages[1] = "[Alice at 2:30 PM]: What about async/await?"
messages[2] = "[GPT-4o] Async/await is a pattern..."
messages[3] = "[Alice at 2:32 PM]: Thanks! Can you show an example?"
4. COLLABORATIVE CONTEXT FILTER (priority: -50)
──────────────────────────────────────────────
Injects participant roster into system message.
messages[0].content += """
## Chat Context (Multi-User Mode)
### Active Participants:
- Alice (owner): Product lead
- Bob (can write): Backend engineer
### Current Request
- Message sender: Alice
- Tools available: calendar, email
"""
5. CONVERSATION CONTEXT FILTER (priority: 50)
────────────────────────────────────────────
Checks token count: 4,500 / 128,000 = 3.5%
Under all thresholds - passes through unchanged.
6. LLM RECEIVES
─────────────
{
"model": "claude-sonnet-4-20250514",
"messages": [
{
"role": "system",
"content": "You are a helpful AI... [multi-model awareness]
## Chat Context (Multi-User Mode)
### Active Participants:
- Alice (owner): Product lead
- Bob (can write): Backend engineer
..."
},
{"role": "user", "content": "[Alice at 2:30 PM]: What about async/await?"},
{"role": "assistant", "content": "[GPT-4o] Async/await is a pattern..."},
{"role": "user", "content": "[Alice at 2:32 PM]: Thanks! Can you show an example?"}
]
}
7. CLAUDE RESPONDS
────────────────
Claude sees that the previous assistant response came from GPT-4o.
It can reference, build upon, or even respectfully correct that response.
Response: "Building on the explanation GPT-4o provided, here's a Python example..."
8. RESPONSE STORED
────────────────
The response is stored in the database with:
- model: "claude-sonnet-4-20250514"
- modelName: "Claude Sonnet 4"
Next time, this message will be tagged [Claude Sonnet 4] for any AI reading it.Lessons Learned
What Worked
- Filter chain modularity: Each filter has a single responsibility. We can add, remove, or reorder filters without breaking others.
- Static base prompts: Prompt caching reduced latency by 40% for repeat users. Keep your system prompt stable!
- Database-backed model names: Storing
modelNameat write time means we don’t need to maintain a mapping of model IDs to display names. - Sticky compression: Once enabled, staying enabled prevents expensive on/off thrashing.
What We Got Wrong (At First)
- Dynamic system prompts: Our first version injected user names, timestamps, and model info directly into the system prompt. This broke prompt caching and added complexity.
- Position-based message matching: We tried matching request messages to DB messages by index. This broke when system messages were added/removed. Solution: track a separate
db_idxthat only increments for user/assistant messages. - Synchronous summarization: Running summarization inline caused timeouts on long conversations. Background tasks with retry logic solved this.
- Tool hallucination: Without explicit tool scoping, AIs would hallucinate access to tools they’d seen in history. The
CRITICAL: Tool Usage Rulessection fixed this.
Things We’re Still Figuring Out
- Cross-model reasoning chains: When Claude references something GPT said, should we track these attribution chains for debugging?
- Model capability awareness: Can we help AIs understand what other models are good at? (“GPT-4o gave you a code example; I can help explain the concepts.”)
- Conflict resolution: When two AIs contradict each other, what’s the best UX for helping users navigate this?
Conclusion
Making multiple AI models aware of each other in group conversations isn’t rocket science, but it requires careful attention to detail. The filter chain pattern gave us the modularity to tackle each challenge independently:
- Base prompt: Teach AIs the rules of multi-model interaction
- Attribution: Tag messages with their true sources
- Context: Provide situational awareness about who’s participating
- Compression: Handle the inevitable growth of long conversations
The result is conversations where Claude can meaningfully build on GPT’s ideas, acknowledge when it’s correcting another model, and understand when tools are available vs. when they’re not.
If you’re building multi-AI features into your own product, the key insight is this: AIs don’t magically understand multi-model contexts. You have to explicitly tell them. A well-crafted system prompt and consistent message attribution go a long way.
