Getting stuck on operations in the frontend (moving stages)

[ch9hn]

Bug Description

When sorting stages in the frontend, the system is getting stuck from time to time.
Only multiple restarts of the containers are solving it.
Its NOT related to upgrades, as the system with this version v1.18.0 is running since three weeks now.
Workspace Version is the same as the container version (v1.18.0).

Expected behavior

It does not get stuck and gives reaction on reload.

Technical inputs

Version

Container Version: twentycrm/twenty:v1.18.1
Workspace Version: 1.18.1

psql "$PG_DATABASE_URL" -c "SELECT id, \"version\", \"activationStatus\" FROM core.\"workspace\";"
                  id                  | version | activationStatus 
--------------------------------------+---------+------------------
 af207be3-9c95-466f-b928-1d99258cd067 | 1.18.1  | ACTIVE
(1 row)

Server Logs:

[EntityBuilder viewGroup] entity processing: 7.003ms
[EntityBuilder viewGroup] validateAndBuild: 8.06ms
[Runner] Initial cache retrieval: 0.035ms
(node:1) Warning: Label '[Runner] Total execution' already exists for console.time()
(node:1) Warning: Label '[Runner] Transaction execution' already exists for console.time()
[BaseWorkspaceMigrationRunnerActionHandlerService] update_viewGroup executeForWorkspaceSchema: 0.141ms
(node:1) Warning: Label '[BaseWorkspaceMigrationRunnerActionHandlerService] update_viewGroup executeForMetadata' already exists for console.time()
[BaseWorkspaceMigrationRunnerActionHandlerService] update_viewGroup executeForMetadata: 24.192ms
(node:1) Warning: No such label '[BaseWorkspaceMigrationRunnerActionHandlerService] update_viewGroup executeForMetadata' for console.timeEnd()
(node:1) Warning: No such label '[BaseWorkspaceMigrationRunnerActionHandlerService] update_viewGroup executeForMetadata' for console.timeEnd()
(node:1) Warning: No such label '[BaseWorkspaceMigrationRunnerActionHandlerService] update_viewGroup executeForMetadata' for console.timeEnd()
(node:1) Warning: No such label '[BaseWorkspaceMigrationRunnerActionHandlerService] update_viewGroup executeForMetadata' for console.timeEnd()
(node:1) Warning: No such label '[BaseWorkspaceMigrationRunnerActionHandlerService] update_viewGroup executeForMetadata' for console.timeEnd()
(node:1) Warning: No such label '[BaseWorkspaceMigrationRunnerActionHandlerService] update_viewGroup executeForMetadata' for console.timeEnd()
[BaseWorkspaceMigrationRunnerActionHandlerService] update_viewGroup executeForWorkspaceSchema: 0.138ms
[BaseWorkspaceMigrationRunnerActionHandlerService] update_viewGroup executeForMetadata: 0.307ms
(node:1) Warning: No such label '[BaseWorkspaceMigrationRunnerActionHandlerService] update_viewGroup executeForMetadata' for console.timeEnd()
(node:1) Warning: No such label '[BaseWorkspaceMigrationRunnerActionHandlerService] update_viewGroup executeForMetadata' for console.timeEnd()
[Runner] Transaction execution: 49.137ms
(node:1) Warning: No such label '[Runner] Transaction execution' for console.timeEnd()
(node:1) Warning: Label '[Runner] Cache invalidation flatViewGroupMaps,flatViewMaps' already exists for console.time()
(node:1) Warning: No such label '[Runner] Transaction execution' for console.timeEnd()
(node:1) Warning: Label '[Runner] Cache invalidation flatViewGroupMaps,flatViewMaps' already exists for console.time()
(node:1) Warning: No such label '[Runner] Transaction execution' for console.timeEnd()
(node:1) Warning: Label '[Runner] Cache invalidation flatViewGroupMaps,flatViewMaps' already exists for console.time()
(node:1) Warning: No such label '[Runner] Transaction execution' for console.timeEnd()
(node:1) Warning: Label '[Runner] Cache invalidation flatViewGroupMaps,flatViewMaps' already exists for console.time()
(node:1) Warning: No such label '[Runner] Transaction execution' for console.timeEnd()
(node:1) Warning: Label '[Runner] Cache invalidation flatViewGroupMaps,flatViewMaps' already exists for console.time()
(node:1) Warning: No such label '[Runner] Transaction execution' for console.timeEnd()
(node:1) Warning: Label '[Runner] Cache invalidation flatViewGroupMaps,flatViewMaps' already exists for console.time()
(node:1) Warning: No such label '[Runner] Transaction execution' for console.timeEnd()
(node:1) Warning: Label '[Runner] Cache invalidation flatViewGroupMaps,flatViewMaps' already exists for console.time()
(node:1) Warning: No such label '[Runner] Transaction execution' for console.timeEnd()
(node:1) Warning: Label '[Runner] Cache invalidation flatViewGroupMaps,flatViewMaps' already exists for console.time()
(node:1) Warning: No such label '[Runner] Transaction execution' for console.timeEnd()
(node:1) Warning: Label '[Runner] Cache invalidation flatViewGroupMaps,flatViewMaps' already exists for console.time()

[sonarly]

🟠 Sonarly Analysis — High Severity

Concurrent viewGroup mutations cause race conditions and server stall

When reordering stages (view groups) in the Kanban board, the frontend fires all viewGroup update mutations concurrently via Promise.all, causing race conditions in the workspace migration runner's cache invalidation, database contention, and a thundering herd effect that stalls the server.

Root Cause

Proximate Cause: Concurrent View Group Mutations Without Serialization

When a user reorders stages in a Kanban view, the frontend's usePerformViewGroupAPIPersist hook fires all viewGroup update mutations concurrently using Promise.all:

// packages/twenty-front/src/modules/views/hooks/internal/usePerformViewGroupAPIPersist.ts:37-43
const results = await Promise.all(
  updateViewGroupInputs.map((variables) =>
    updateViewGroupMutation({
      variables,
    }),
  ),
);

Each mutation independently goes through the full workspace migration pipeline:

1. ViewGroupService.updateOne → validateBuildAndRunWorkspaceMigration
2. WorkspaceMigrationRunnerService.run — creates a database transaction, executes the update, commits
3. WorkspaceMigrationRunnerService.invalidateCache — invalidates and recomputes flat entity map caches

The server logs from the issue confirm concurrent execution — console.time label collision warnings prove multiple run() calls are in-flight simultaneously:

Warning: Label '[Runner] Total execution' already exists for console.time()
Warning: Label '[Runner] Transaction execution' already exists for console.time()
Warning: Label '[Runner] Cache invalidation flatViewGroupMaps,flatViewMaps' already exists

There is no concurrency control (mutex, lock, or queue) in the migration runner path.

The Race Condition

The invalidateAndRecompute method in WorkspaceCacheService (line 163-171) performs three non-atomic steps:

async invalidateAndRecompute(workspaceId, cacheKeyNames) {
  await this.memoizer.clearKeys(`${workspaceId}-`);  // Step 1: clear memoizer
  await this.flush(workspaceId, cacheKeyNames);        // Step 2: flush Redis
  await this.recomputeDataFromProvider(workspaceId, cacheKeyNames); // Step 3: recompute
}

When N concurrent viewGroup updates all reach this path:

1. Request A clears the memoizer cache for the workspace
2. Request B (still in its getOrRecompute for pre-transaction setup) finds empty cache, triggers a full recomputation
3. Request C also triggers recomputation
4. All N requests simultaneously recompute the flat entity maps from the database, creating a thundering herd
5. Each recomputation reads the database, which may be under contention from other concurrent transactions

The PromiseMemoizer does coalesce concurrent identical promise keys (line 34-37 of promise-memoizer.storage.ts), but clearKeys (called by invalidateAndRecompute) deletes pending promises too (line 91-95), destroying this protection.

Why It Gets Stuck

A Kanban board with N stages fires N concurrent mutations. Each goes through the full migration runner pipeline independently. The combination of:

• PostgreSQL row-level lock contention on viewGroup rows being updated simultaneously
• Cache thundering herd from repeated invalidation/recomputation cycles
• No serialization of mutations that logically should be a single batch operation

...causes the server to become overwhelmed. The GraphQL requests from the frontend hang indefinitely, making the UI appear stuck. Only container restarts clear the accumulated state (stuck transactions, corrupted cache entries).

Triggering Cause

This is triggered by any user reordering stages in a Kanban view when the view has multiple stages. The reorderRecordGroups function in useReorderRecordGroups.ts calls saveViewGroups which sends position updates for all affected groups simultaneously. The more stages being reordered, the more concurrent mutations, and the higher the probability of hitting the race condition. The intermittent nature ("from time to time") aligns with this being a timing-dependent race — it depends on how many stages are affected and how the concurrent requests interleave on the server.

Suggested Fix

Frontend fix (root cause): Changed usePerformViewGroupAPIPersist to execute viewGroup update mutations sequentially instead of concurrently. The Promise.all pattern fired all N mutations simultaneously, each triggering a full workspace migration runner pipeline (transaction + cache invalidation). The sequential for...of loop ensures each mutation completes (including its cache invalidation) before the next begins, eliminating the race condition that caused server stalls.

Backend monitoring (observability): Added unique run IDs to all console.time labels in the workspace migration runner to prevent label collisions when concurrent executions do occur. Added an activeRunsByWorkspace counter that logs a warning when concurrent migration runner executions are detected for the same workspace, making this class of issue immediately visible in server logs.

---

Analyzed by Sonarly · View full analysis

[charlesBochet]

Ok, I still think as it's not something we have observed on the cloud. It's likely a combination of small instance on your side and inefficiency on Twenty code.
Could you start by giving more vCPU: at least 1vCPU and 2GB RAM?