Performance/Refactoring: Migrate reconnect message synchronization from DDP loadMissedMessages to REST syncMessages with index optimizations

[Shevilll]

Performance/Refactoring: Migrate reconnect message synchronization from DDP loadMissedMessages to REST syncMessages with index optimizations

Context & Problem Statement

Currently, when a client loses connection and subsequently reconnects to a Rocket.Chat server, the reconnection sync flow loads missed messages using a legacy DDP method: 'loadMissedMessages'.

While this legacy flow has worked historically, it has several critical flaws that undermine data consistency and limit Rocket.Chat's architectural scalability:

1. Missing Offline Edits: The legacy DDP query retrieves messages based on their creation timestamp (ts). If a message is edited by another user while the client is offline, the client will completely miss this update because the creation ts is unchanged, leading to stale data on the client.
2. Missing Offline Deletions: The client is not informed of deletions that occur while offline. This leads to "ghost" messages remaining visible on the client UI until a full application reload is forced.
3. DDP Overhead: This mechanism locks Rocket.Chat into heavy, stateful, WebSocket-based DDP operations, obstructing efforts to migrate entirely to modern, stateless HTTP load balancers and REST-first APIs.

Proposed REST-Based Sync Architecture

To solve these flaws and advance the deprecation of legacy DDP, we propose migrating the reconnection message sync to the lightweight GET /v1/chat.syncMessages REST endpoint. This endpoint is designed to return both updated (edited/created) and deleted messages since a given timestamp in a single query.

Sequence Diagram

sequenceDiagram
    participant Client as Client (Zustand Stores)
    participant Hook as useLoadMissedMessages Hook
    participant API as REST API (/v1/chat.syncMessages)
    participant DB as MongoDB (Message & Trash)

    Client->>Hook: Connection Restored (connected: true)
    Hook->>Client: Retrieve last synced timestamp (lastUpdate)
    Hook->>API: GET /v1/chat.syncMessages?roomId={rid}&lastUpdate={lastUpdate}
    API->>DB: Query updated & deleted messages since lastUpdate
    DB-->>API: Returns { updated: IMessage[], deleted: Array }
    API-->>Hook: Return JSON Response
    Hook->>Client: Upsert updated messages (upsertMessageBulk)
    Hook->>Client: Purge deleted messages (Messages.state.delete)

Loading

Proposed Client-Side Hook (useLoadMissedMessages.ts)

import type { IRoom } from '@rocket.chat/core-typings';
import { useConnectionStatus, useEndpoint } from '@rocket.chat/ui-contexts';
import { useEffect, useRef } from 'react';

import { LegacyRoomManager, upsertMessageBulk } from '../../../../app/ui-utils/client';
import { Messages, Subscriptions } from '../../../stores';

/**
 * Synchronizes missed, edited, and deleted messages for a room using REST API.
 */
const syncRoomMessages = async (
	rid: IRoom['_id'],
	syncMessages: (params: { roomId: string; lastUpdate: string }) => Promise<{
		result: {
			updated: any[];
			deleted: { _id: string; _deletedAt: string }[];
		};
		success: boolean;
	}>
): Promise<void> => {
	// Find the latest updated message to establish a highly reliable synchronization epoch
	const lastMessage = Messages.state.findFirst(
		(record) => record.rid === rid && record._hidden !== true && !record.temp,
		(a, b) => b._updatedAt.getTime() - a._updatedAt.getTime(),
	);

	if (!lastMessage) {
		return;
	}

	try {
		// Fetch updates and deletions since lastMessage._updatedAt
		const response = await syncMessages({
			roomId: rid,
			lastUpdate: lastMessage._updatedAt.toISOString(),
		});

		if (response.success && response.result) {
			const { updated, deleted } = response.result;
			const subscription = Subscriptions.state.find((record) => record.rid === rid);

			// 1. Reconcile added and edited messages
			if (updated && updated.length > 0) {
				await upsertMessageBulk({ msgs: updated, subscription });
			}

			// 2. Reconcile deleted messages
			if (deleted && deleted.length > 0) {
				deleted.forEach((delRecord) => {
					Messages.state.delete(delRecord._id);
				});
			}
		}
	} catch (error) {
		console.error('Error synchronizing room messages:', error);
	}
};

export const useLoadMissedMessages = (): void => {
	const { connected } = useConnectionStatus();
	const connectionWasOnlineRef = useRef(connected);
	const syncMessages = useEndpoint('GET', '/v1/chat.syncMessages');

	useEffect(() => {
		if (connected === true && connectionWasOnlineRef.current === false && LegacyRoomManager.openedRooms) {
			Object.keys(LegacyRoomManager.openedRooms).forEach((key) => {
				const value = LegacyRoomManager.openedRooms[key];
				if (value.rid) {
					syncRoomMessages(value.rid, syncMessages);
				}
			});
		}

		connectionWasOnlineRef.current = connected;
	}, [connected, syncMessages]);
};

---

Database Performance Analysis & Index Optimization

The codebase currently notes that /v1/chat.syncMessages is "currently too slow for this path" (see useLoadMissedMessages.ts:L24-26). Let's dissect the database query pattern to resolve this:

The Query Pattern of chat.syncMessages

The server-side implementation in apps/meteor/server/publications/messages.ts delegates to handleWithoutPagination(rid, lastUpdate):

const [updatedMessages, deletedMessages] = await Promise.all([
    Messages.findForUpdates(rid, query, options).toArray(),
    Messages.trashFindDeletedAfter(lastUpdate, { rid }, { projection: { _id: 1, _deletedAt: 1 }, ...options }).toArray(),
]);

1. For Updated Messages:

   • Filter: { rid: roomId, _hidden: { $ne: true }, _updatedAt: { $gt: lastUpdate } }
   • Current Index: { rid: 1, ts: 1, _updatedAt: 1 }
   • Performance Bottleneck: Because ts is positioned between rid and _updatedAt, MongoDB must scan every index key matching rid to evaluate the range query on _updatedAt. This behaves as an $O(N)$ scan of the channel's message index keys, scaling extremely poorly for large channels with high volume.
   • Optimized Solution: Create a dedicated compound index on the message collection:

     { rid: 1, _updatedAt: 1 }

     This allows MongoDB to perform an exact boundary scan on _updatedAt for the target roomId, ensuring $O(\log N)$ complexity.

2. For Deleted Messages (Trash Collection):

   • Filter: { __collection__: 'message', _deletedAt: { $gt: lastUpdate }, rid: roomId }
   • Current Index: No specific compound index is defined for rid and _deletedAt in the trash collection, causing slow collection-wide scans on delete audits.
   • Optimized Solution: Create a compound index on the trash collection:

     { __collection__: 1, rid: 1, _deletedAt: 1 }

Automated Index Application

To apply the optimized index on startup, add it to modelIndexes() in packages/models/src/models/Messages.ts:

protected override modelIndexes(): IndexDescription[] {
	return [
		{ key: { rid: 1, ts: 1, _updatedAt: 1 } },
		{ key: { rid: 1, _updatedAt: 1 } },
		{ key: { ts: 1 } },
		...

---

Action Items & Migration Roadmap

1. Optimize Database Indexes: Modify packages/models/src/models/Messages.ts to include { rid: 1, _updatedAt: 1 }.
2. Implement Client REST Sync: Refactor the useLoadMissedMessages hook to invoke /v1/chat.syncMessages and reconcile both updated and deleted message lists directly in Zustand state.
3. Handle Edge Cases:
   • Verify e2e rooms (handling encryption/decryption keys and pending statuses).
   • Ensure handling of potential large batches by checking if pagination is needed should the number of offline changes exceed a threshold.
4. Deprecate Legacy Code: Safely remove apps/meteor/server/methods/loadMissedMessages.ts and clean up its registration.

[coderabbitai]

⚠️ Possible Duplicate Issue(s)

• (ddp-removal): this should move to /v1/chat.syncMessages so reconnect #41000

🔗 Related PRs

#40659 - chore: migrate DDP callers to REST endpoints (used methods with REST replacement) [merged]
#40954 - chore: reduce room opening latency in message history load [merged]
#40981 - chore: undo loadMissedMessages deprecation from #40711 [merged]
#40991 - fix: retry transient errors when opening a room [merged]
#40996 - feat: add rooms.join endpoint for all room types [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!