0.5.x Extremely slow server performance

[joaofernandes-fedrax]

Gorse version
0.5.4

Describe the bug
There's a huge difference in the API performance between version 0.4.x and 0.5.x.

To Reproduce
New setup: Gorse (master, worker, server) 0.5.4, Redis 8.6.0, MySQL 8.
Old setup: Gorse (master, worker, server) 0.4.16, Redis 8.6.0, MySQL 8.

Version 0.4.16 handles 300 rps on the API (POST /feedback and GET /recommend/xyz) with 3ms response time, while training.
Version 0.5.4 slows down to 900ms on POST /feedback, and 15s GET /recommend/xyz while training. Drops to normal 3ms when stopping "master" container.

Expected behavior
Performance should be similar across versions.

Additional context

• Server A: Gorse + Redis server with 88 cores, 1TB RAM
• Server B: MySQL Server with 56 cores, 750GB RAM
• About 12M feedbacks and 400k users per day. 40k items, +300 per day.
• Most settings on config.toml are set to default, except n_jobs, periods and cache

[zhenghaoz]

Drops to normal 3ms when stopping "master" container.

Have you noticed CPU usage of MySQL server? I suspect it casued by some heavy write from the master.

[joaofernandes-fedrax]

No abnormal CPU usage on the MySQL server or any stuck queries - only some slow queries during initial dataset load taking up to 40s when importing feedback but nothing else after.

The biggest difference in usage is on the Redis server. Even after setting io-threads=16 it seems to be the bottleneck, with many HSET commands taking 30ms (from SLOWLOG).

POST /items is the worst part right now, many requests are taking minutes to finish (1 item per request):

"msg":"POST /api/items","status_code":200,"response_time":76.869962635
"msg":"POST /api/items","status_code":200,"response_time":14.388010042
"msg":"POST /api/items","status_code":200,"response_time":124.752710937
"msg":"POST /api/items","status_code":200,"response_time":185.010736277
"msg":"POST /api/items","status_code":200,"response_time":0.007695327
"msg":"POST /api/items","status_code":200,"response_time":452.695257028
"msg":"POST /api/items","status_code":200,"response_time":558.108063333
"msg":"POST /api/items","status_code":200,"response_time":0.023342459

I've disabled [recommend.ranker] and set to only "user-to-user/neighbors" to speed up recommendations as some were taking up to 30s to respond with the default config.
Each GET /recommend/xyz now takes on average 500ms to complete, doubling on some cases, depending on the number of feedbacks for the user.

Right now I'm sitting at 40k items, 1M users and 45M feedbacks.

I'm aware this is an open-source project and you're under no obligation to help with my specific case.
Thank you in advance!