Answer process jumps after few days to ~ 99% CPU usage

[brocaar]

Describe the bug

After leaving Answer for some time (days?) running, the CPU usage jumps and stays at around 99%, even when it is not serving any requests. The web-interface is still responsive enough to keep it functional, but after a restart (I have to kill the process) it hardly consumes any CPU.

To Reproduce

I'm not sure how to reproduce / trigger this. It happens after a couple of days running. I noticed web-requests take slightly longer and I inspected the CPU usage. This already happened a few times in a row.

Expected behavior

CPU usage stays stable.

Logs

Feb 02 10:32:48 chirpstack-support answer[2067003]: 2026-02-02 10:32:48.045        ERROR        activity_common/vote.go:60        pq: invalid input syntax for type bigint: ""
Feb 02 10:32:48 chirpstack-support answer[2067003]: 2026-02-02 10:32:48.288        ERROR        activity_common/vote.go:60        pq: invalid input syntax for type bigint: ""
Feb 02 10:32:48 chirpstack-support answer[2067003]: 2026-02-02 10:32:48.429        ERROR        activity_common/vote.go:60        pq: invalid input syntax for type bigint: ""
Feb 02 10:36:25 chirpstack-support answer[2067003]: 2026-02-02 10:36:25.605        ERROR        activity_common/vote.go:60        pq: invalid input syntax for type bigint: ""
Feb 02 10:36:25 chirpstack-support answer[2067003]: 2026-02-02 10:36:25.767        ERROR        activity_common/vote.go:60        pq: invalid input syntax for type bigint: ""
Feb 02 10:40:00 chirpstack-support answer[2067003]: 2026-02-02 10:40:00.025        INFO        cron/cron.go:88        checking expired user suspensions
Feb 02 10:45:16 chirpstack-support answer[2067003]: 2026-02-02 10:45:16.946        ERROR        activity_common/vote.go:60        pq: invalid input syntax for type bigint: ""
Feb 02 10:45:31 chirpstack-support answer[2067003]: 2026-02-02 10:45:31.729        ERROR        activity_common/vote.go:60        pq: invalid input syntax for type bigint: ""
Feb 02 10:45:31 chirpstack-support answer[2067003]: 2026-02-02 10:45:31.870        ERROR        activity_common/vote.go:60        pq: invalid input syntax for type bigint: ""
Feb 02 10:45:32 chirpstack-support answer[2067003]: 2026-02-02 10:45:32.032        ERROR        activity_common/vote.go:60        pq: invalid input syntax for type bigint: ""

There is nothing special in the logs. I will restart the process with LOG_LEVEL=DEBUG to gather more log-output.

Plugins

Platform

• OS: Linux AMD64
• Version: v1.7.1

[LinkinStars]

@brocaar Alright, if any additional error logs appear, please provide them to us. Then, please share the list of plugins you're currently using, and we'll continue to follow up on this issue.

[brocaar]

@LinkinStars, will do. I just restarted the process with DEBUG logging. It is now running at ~ 1% CPU as it should. Once it starts spiking again, I'll let you know.

[brocaar]

@LinkinStars it is back at 99% CPU. However, I can not find anything in the logs that might point to the cause....

The only additional logs I see with DEBUG are:

DEBUG        router/ui.go:68        open static path build/static/js/ .....

and

DEBUG        rank/rank_service.go:255        user XXX want to do action ....

I see you released a v2, which also solves two other issues that I reported. I will try upgrade to that version and continue my testing.

From the logs, I see there is a kind of cron scheduler that schedules periodic tasks. I have not looked at the implementation, but could it be that one of these tasks gets stuck, keeps retrying, then the scheduler schedules a new duplicate cron task meaning that there are now two tasks retrying, shortly after three and so on...?

[LinkinStars]

From the logs, I see there is a kind of cron scheduler that schedules periodic tasks. I have not looked at the implementation, but could it be that one of these tasks gets stuck, keeps retrying, then the scheduler schedules a new duplicate cron task meaning that there are now two tasks retrying, shortly after three and so on...?

It's unlikely caused by scheduled tasks. If it were, our tests should have detected it. We suspect it's triggered under specific conditions. We've deployed pprof and prometheus to monitor CPU and memory usage. Should any issues arise, we'll fix them ASAP.

[brocaar]

@LinkinStars just to let you know, with v2, I still this behavior once leaving it running overnight... What else can I do to help you debugging this? It seems it takes ~ 24 hours before I see this, e.g. I notice in the morning, I restart it and during the day all is fine. It does not seem to slowly rise. Then the next morning when I check again it is at ~99%.

[brocaar]

@LinkinStars what would be the steps to re-compile answer with code modifications (e.g. adding pprof)? I can make Go code changes, but since I'm using the answer build command, it seems there is some magic being done automatically (fetching the source, compiling the frontend, and including the plugins in the answer binary).

[LinkinStars]

@brocaar If possible, you can build a binary or Docker image using the dev branch and package the plugins that you need for testing. If the issue persists, please continue to provide feedback to me. Thank you.

[brocaar]

@LinkinStars , could you confirm the steps for this?

Should I follow these steps first:
https://github.com/apache/answer?tab=readme-ov-file#build

And then using that binary execute ./answer build .... to build the binary with plugins? Or is there also an option to include the plugins directly when building from source?

[LinkinStars]

Should I follow these steps first: https://github.com/apache/answer?tab=readme-ov-file#build

And then using that binary execute ./answer build .... to build the binary with plugins

@brocaar Yes. Your steps are correct. The ANSWER_MODULE environment variable must be set.

make ui
make build

export ANSWER_MODULE="$PWD"
./answer build \
  --output ./new_answer \
  --with github.com/apache/answer-plugins/cache-redis@latest \
  --with github.com/apache/answer-plugins/connector-github@latest \
  --with github.com/apache/answer-plugins/connector-google@latest \
  --with github.com/apache/answer-plugins/quick-links@latest \
  --with github.com/apache/answer-plugins/render-markdown-codehighlight@latest \
  --with github.com/apache/answer-plugins/reviewer-basic@latest \
  --with github.com/apache/answer-plugins/storage-s3@latest

[brocaar]

@LinkinStars I will try the above next week, but I found this error this morning in the logs, around the same time there was a CPU spike:

Feb 05 09:44:56 chirpstack-support answer[2147189]: 2026-02-05 09:44:56.921        DEBUG        controller/connector_controller.go:134        connector received: {ExternalID:XXXXXX DisplayName:XXXX XXXXXX Username:XXXXX XXXXX Email:XXXXXXX Avatar:https://lh3.googleusercontent.com/[1000+ chars string] MetaInfo:{
Feb 05 09:44:56 chirpstack-support answer[2147189]:   "sub": "XXXX",
Feb 05 09:44:56 chirpstack-support answer[2147189]:   "name": "XXXXXX",
Feb 05 09:44:56 chirpstack-support answer[2147189]:   "given_name": "XXXXXX ",
Feb 05 09:44:56 chirpstack-support answer[2147189]:   "family_name": "XXXX",
Feb 05 09:44:56 chirpstack-support answer[2147189]:   "picture": "https://lh3.googleusercontent.com/a-/[1000+ chars string]",
Feb 05 09:44:56 chirpstack-support answer[2147189]:   "email": "XXXXX",
Feb 05 09:44:56 chirpstack-support answer[2147189]:   "email_verified": true,
Feb 05 09:44:56 chirpstack-support answer[2147189]:   "hd": "XXXXXXX"
Feb 05 09:44:56 chirpstack-support answer[2147189]: }}
Feb 05 09:44:57 chirpstack-support answer[2147189]: 2026-02-05 09:44:57.646        ERROR        controller/connector_controller.go:146        external login failed: code: 500, reason: base.database_error, message: , error: pq: value too long for type character varying(1024)
Feb 05 09:44:57 chirpstack-support answer[2147189]: /apache-answer-2.0.0-bin-linux-amd64/answer_build3005906725/vendor/github.com/apache/answer/internal/repo/user/user_repo.go:66 github.com/apache/answer/internal/repo/user.(*userRepo).AddUser.func1
Feb 05 09:44:57 chirpstack-support answer[2147189]: /apache-answer-2.0.0-bin-linux-amd64/answer_build3005906725/vendor/xorm.io/xorm/engine.go:1423 xorm.io/xorm.(*Engine).Transaction
Feb 05 09:44:57 chirpstack-support answer[2147189]: /apache-answer-2.0.0-bin-linux-amd64/answer_build3005906725/vendor/github.com/apache/answer/internal/repo/user/user_repo.go:54 github.com/apache/answer/internal/repo/user.(*userRepo).AddUser
Feb 05 09:44:57 chirpstack-support answer[2147189]: /apache-answer-2.0.0-bin-linux-amd64/answer_build3005906725/vendor/github.com/apache/answer/internal/service/user_external_login/user_external_login_service.go:209 github.com/apache/answer/internal/service/user_external_login.(*UserExternalLoginService).registerNewUser
Feb 05 09:44:57 chirpstack-support answer[2147189]: /apache-answer-2.0.0-bin-linux-amd64/answer_build3005906725/vendor/github.com/apache/answer/internal/service/user_external_login/user_external_login_service.go:156 github.com/apache/answer/internal/service/user_external_login.(*UserExternalLoginService).ExternalLogin
Feb 05 09:44:57 chirpstack-support answer[2147189]: /apache-answer-2.0.0-bin-linux-amd64/answer_build3005906725/vendor/github.com/apache/answer/internal/controller/connector_controller.go:144 github.com/apache/answer/internal/controller.(*ConnectorController).ConnectorRedirectDispatcher.(*ConnectorController).ConnectorRedirect.func2
Feb 05 09:44:57 chirpstack-support answer[2147189]: /apache-answer-2.0.0-bin-linux-amd64/answer_build3005906725/vendor/github.com/apache/answer/internal/controller/connector_controller.go:97 github.com/apache/answer/internal/controller.(*ConnectorController).ConnectorRedirectDispatcher
Feb 05 09:44:57 chirpstack-support answer[2147189]: /apache-answer-2.0.0-bin-linux-amd64/answer_build3005906725/vendor/github.com/gin-gonic/gin/context.go:185 github.com/gin-gonic/gin.(*Context).Next
Feb 05 09:44:57 chirpstack-support answer[2147189]: /apache-answer-2.0.0-bin-linux-amd64/answer_build3005906725/vendor/github.com/anargu/gin-brotli/gin_brotli.go:71 github.com/apache/answer/internal/base/server.NewHTTPServer.Brotli.func3
Feb 05 09:44:57 chirpstack-support answer[2147189]: /apache-answer-2.0.0-bin-linux-amd64/answer_build3005906725/vendor/github.com/gin-gonic/gin/context.go:185 github.com/gin-gonic/gin.(*Context).Next
Feb 05 09:44:57 chirpstack-support answer[2147189]: /apache-answer-2.0.0-bin-linux-amd64/answer_build3005906725/vendor/github.com/gin-gonic/gin/gin.go:633 github.com/gin-gonic/gin.(*Engine).handleHTTPRequest
Feb 05 09:44:57 chirpstack-support answer[2147189]: /apache-answer-2.0.0-bin-linux-amd64/answer_build3005906725/vendor/github.com/gin-gonic/gin/gin.go:589 github.com/gin-gonic/gin.(*Engine).ServeHTTP
Feb 05 09:44:57 chirpstack-support answer[2147189]: /nix/store/4wc5q32wgrhac7vb4da1zclhvw7r01wx-go-1.25.5/share/go/src/net/http/server.go:3340 net/http.serverHandler.ServeHTTP
Feb 05 09:44:57 chirpstack-support answer[2147189]: /nix/store/4wc5q32wgrhac7vb4da1zclhvw7r01wx-go-1.25.5/share/go/src/net/http/server.go:2109 net/http.(*conn).serve
Feb 05 09:44:57 chirpstack-support answer[2147189]: /nix/store/4wc5q32wgrhac7vb4da1zclhvw7r01wx-go-1.25.5/share/go/src/runtime/asm_amd64.s:1693 runtime.goexit

Even if it turns out it is not related, then I think it still should be fixed.... I have created a separate issue for this.

Edit: I don't see this error around the other CPU spikes....

[maishivamhoo123]

@LinkinStars and @team can you please assign this issue to me? I love to contribute to this isssue.