Crash due to unknown string encoding

[edmcman]

I'm getting a crash when I run codeql database analyze my-db codeql/cpp-queries:codeql-suites/cpp-security-extended.qls --format sarif-latest --output results.sarif:

Oops! A fatal internal error occurred. Details:
com.semmle.inmemory.DBError: Unknown string encoding: 4389148. Maximum current encoding is: 4387101. There are 704544 extensional strings,643554 cached strings and 3682557 intensional strings.
    at com.semmle.inmemory.stringpool.StringPool.noSuchStringError(StringPool.java:536)
    at com.semmle.inmemory.stringpool.StringPool.getBinaryEncodingOfString(StringPool.java:689)
    at com.semmle.inmemory.relations.BinaryQueryResultSetsEncoder$StringPoolQueryResultsBuilder.encodeString(BinaryQueryResultSetsEncoder.java:99)
    at com.semmle.inmemory.relations.BinaryQueryResultSetsEncoder$Encoder$4.encode(BinaryQueryResultSetsEncoder.java:44)
    at com.semmle.inmemory.relations.BinaryQueryResultSetsEncoder$BinaryQueryResultSetEncoder.addTuple(BinaryQueryResultSetsEncoder.java:169)
    at com.semmle.inmemory.relations.BaseIntArrayRelation.map(BaseIntArrayRelation.java:88)
    at com.semmle.inmemory.caching.PagedRelation.map(PagedRelation.java:158)
    at com.semmle.inmemory.MemoryBackend.getResultSets(MemoryBackend.java:258)
    at com.semmle.inmemory.MemoryBackend$QueryOrchestrator.lambda$startEverything$1(MemoryBackend.java:486)
    at com.semmle.inmemory.scheduler.CapstoneTask.evaluate(CapstoneTask.java:178)
    at com.semmle.inmemory.scheduler.CapstoneTask$2.doWork(CapstoneTask.java:168)
    at com.semmle.inmemory.scheduler.execution.ThreadableWork.doSomeWork(ThreadableWork.java:396)
    at com.semmle.inmemory.scheduler.execution.ExecutionScheduler.runnerMain(ExecutionScheduler.java:713)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)

It takes about an hour to trigger. So far, it has happened 2/2 attempts.

This is the repository I am analyzing: https://github.com/arichardson/juliet-test-suite-c/

I'm using codeql 2.25.0.

I'm happy to upload more data/information; just tell me what you need.

[jketema]

Hi @edmcman

Thanks for the report. Can you share how you created the database, and on which platform?

[edmcman]

Sure, I ran ./juliet.py -ga and then codeql database create my-db --language=cpp --command="./juliet.py -ma".

I had some trouble with my compilers (clang-18) defaulting to too recent standards, so I also have the following patch:

diff --git a/juliet.py b/juliet.py
index b4a26e84a..9a3976e6d 100755
--- a/juliet.py
+++ b/juliet.py
@@ -24,7 +24,7 @@ def clean(path):
 
 def generate(path, output_dir, keep_going):
     shutil.copy(root_dir + "/CMakeLists.txt", path)
-    retcode = subprocess.Popen(["cmake", "-DOUTPUT_DIR:STRING=" + output_dir, "."], cwd=path).wait()
+    retcode = subprocess.Popen(["cmake", "-DCMAKE_C_STANDARD=99", "-DCMAKE_CXX_STANDARD=14", "-DOUTPUT_DIR:STRING=" + output_dir, "."], cwd=path).wait()
     if retcode != 0 and not keep_going:
         juliet_print("error generating " + path + " - stopping")
         exit()

ed@banana ~/P/juliet-test-suite-c (master)> lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.5 LTS
Release:	22.04
Codename:	jammy
ed@banana ~/P/juliet-test-suite-c (master)> echo $CC
clang-18
ed@banana ~/P/juliet-test-suite-c (master)> echo $CXX
clang++-18

[edmcman]

I think running with --threads 1 worked? It did print this at the end:

Shutting down query evaluator.
Interpreting results.
Will not interpret file coverage baseline information, since the total number of baseline files is 105191, which is greater than the limit of 50000.
CodeQL scanned 68723 out of 105188 C/C++ files in this invocation. Typically CodeQL is configured to analyze a single CodeQL language per invocation, so check other invocations to determine overall coverage information.

[jketema]

I think running with --threads 1 worked?

--threads 1 is the default. So unless you specified it explicitly before that should not have changed anything.

[edmcman]

OK, probably just non-determinism rearing it's head. I'll try with --threads 1 and few more times to verify.

[jketema]

OK, probably just non-determinism rearing it's head. I'll try with --threads 1 and few more times to verify.

Thanks. I'm currently also looking locally if I can reproduce the problem.

[edmcman]

Yup, confirmed it still happens with --threads 1. I also rebuilt the database for good measure (problem still happens).