S3EncryptionClient (not async) Unable to execute HTTP request: headersFuture is still not completed when onStream() is invoked.

[EvgenyKislyy]

The code where I create S3EncryptionClient and call it (3.1.1 library version)

  private S3EncryptionClient getEncryptionClient(String regionName, KeyPair keyPair) {
        return S3EncryptionClient.builder()
                .rsaKeyPair(keyPair)
                .enableDelayedAuthenticationMode(true)
                .enableLegacyUnauthenticatedModes(true)
                .enableLegacyWrappingAlgorithms(true)
                .wrappedClient(getClient(regionName))
                .wrappedAsyncClient(getAsyncClient(regionName))
                .build();
    }
    ...
       private S3AsyncClient getAsyncClient(String regionName) {
        return S3AsyncClient.builder()
                .overrideConfiguration(createClientConfiguration())
                .region(Region.of(regionName))
                .build();
    } 
    ..
        private ClientOverrideConfiguration createClientConfiguration() {
        return S3Client.builder()
                .httpClient(
                        ApacheHttpClient.builder()
                                .connectionTimeout(Duration.ofMillis(clientConnectionTimeout))
                                .build())
                .overrideConfiguration()
                .toBuilder()
                .retryPolicy(RetryPolicy.builder().numRetries(NUM_RETRIES).build())
                .apiCallAttemptTimeout(Duration.ofMillis(clientRequestTimeout))
                .build();
    }
...
...
     return getInputStream(s3Client.getObject(getGetObjectRequest(storageKey)), isZip);
 ...
     private InputStream getInputStream(ResponseInputStream<GetObjectResponse> is, boolean isZip)
            throws IOException {
        InputStream s3InputStream;
        if (isZip) {
            ZipInputStream zis = new ZipInputStream(is);
            ZipEntry zipEntry = zis.getNextEntry(); // There should be only one entry in the ZIP file

            log.debug("getReader: Found zip entry {}", zipEntry.getName());

            s3InputStream = zis;
        } else {
            s3InputStream = is;
        }
        return s3InputStream;
    }

The error I see:

Caused by: sotware.amazon.encryption.s3.S3EncryptionClientException: Unable to execute HTTP request: headersFuture is still not completed when onStream() is invoked.
	at deployment.fs.war//software.amazon.encryption.s3.S3EncryptionClient.getObject(S3EncryptionClient.java:255)
	
	Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: headersFuture is still not completed when onStream() is invoked.
	at deployment.fs.war//software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
	at deployment.fs.war//software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
	at deployment.fs.war//software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:223)
	at deployment.fs.war//software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:218)
	at deployment.fs.war//software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:182)
	at deployment.fs.war//software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
	at deployment.fs.war//software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
	at deployment.fs.war//software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:108)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
	at deployment.fs.war//software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:255)
	at deployment.fs.war//software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167)
	at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930)
	at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907)
	at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

The question is - why do I see something about the future if I do not use an async client?
How can I correctly get the response and use InputStream here? I can't see any wait/join function to wait

[EvgenyKislyy]

If you see this strange error, maybe it means that you are trying to get an unecrypted object with an Encrypted Client. In my case it helps

Would be better to see the more clear error message

Previously (in the v1 of was SDK) it was Unable to detect encryption information for object 'key' in bucket 'bucket'. Returning object without decryption.

[texastony]

@EvgenyKislyy

Thanks for cutting the issue.

The Amazon S3 Encryption Client (S3EC) always uses
an Async client to for Get or Put object.

It appears you have found a bug,
where the getObject request fails because the object is not encrypted,
but the clean up of async resources gets interrupted,
resulting in an unhelpful error message.

Can you confirm the object was not encrypted?

If that is the case,
we can create a new test case that recreates this scenario,
and then attempt to patch this.

[EvgenyKislyy]

@texastony Yes, it was an attempt to read unencrypted objects with the encrypted client (as I understood after couple of days of investigation),
it looks like before, for AWS S3 Encryption Client V2 we used withUnsafeUndecryptableObjectPassthrough

AmazonS3EncryptionClientV2Builder.standard()
        .withCryptoConfiguration(
                new CryptoConfigurationV2(CryptoMode.AuthenticatedEncryption)
                        .withUnsafeUndecryptableObjectPassthrough(true))
                .withEncryptionMaterialsProvider(materials)
                .withClientConfiguration(createClientConfiguration())
                .withRegion(regionName)
                .build()

But in the current version, we can't see this option, and we see this error happening.

[texastony]

@EvgenyKislyy

Alas, the AWS S3 Encryption Client V3 (S3EC) does not support reading plaintext objects.
A normal S3 Client MUST be used to read plaintext objects.

I will cut a PR to call this out in the Migration section of the README.

We already have a test that reads plaintext objects with the S3EC.
I ran that locally, the error message is not what you have shown us.

So our test is not re-producing your result,
we are missing something.

Do you mind sharing how large this file is, roughly?

A colleague of mine has suggested this could be caused by the S3EC not finding metadata or an Instruction File,
but that is also tested and should throw a different error.

Open AIs:

• Reproduce headersFuture exception, possibly by running a plaintext test with enableDelayedAuthenticationMode and legacy modes enabled, and with a larger than file.
• Update the README to declare withUnsafeUndecryptableObjectPassthrough as unsupported in V3

[EvgenyKislyy]

It's a small file under 1mb usually (for example, a small JSON in a zip archive, with the attached metadata)

Or some small json/txt file without any archiving

Is there any way to understand is it encrypted object or not? If I have the key and the bucket
Currently we have some kind of stupid logic
try to read the file with the encrypted client, if we see some encriptionclientexception, then try to read it with unencrypted client

we are started to see this bug after migration from AWS-SDK v1 to AWS-SDK v2