blob/sftpblob: add sftpblob driver for SFTP-based blob storage support#3682
Closed
TuSKan wants to merge 2 commits intogoogle:masterfrom
Closed
blob/sftpblob: add sftpblob driver for SFTP-based blob storage support#3682TuSKan wants to merge 2 commits intogoogle:masterfrom
TuSKan wants to merge 2 commits intogoogle:masterfrom
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
…for direct PEM byte parsing
Contributor
|
See comment on the Issue. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR introduces the
sftpblobdriver, allowinggocloud.dev/blobto interface with SFTP servers.SFTP remains a critical protocol for legacy enterprise integrations and B2B data pipelines. This driver bridges the gap, allowing users to interact with SFTP endpoints using the standard Go CDK blob interface.
Implementation Details
Underlying Libraries: Utilizes
github.com/pkg/sftpfor the protocol implementation andgolang.org/x/crypto/sshfor connection management.Authentication: Supports password, private key (PEM), and
SSH_AUTH_SOCKagent authentication via URL parameters.Atomic Writes: To prevent partial uploads from corrupting data, writes are streamed to a temporary file (
.tmp). The driver tracksWritestate, and only executes aPosixRenameto the final path upon a successfulClose(). Failed writes automatically clean up the temporary file.Context Management: Decouples the setup context used in
OpenBucketURLfrom the ongoing connection lifecycle to prevent background goroutine panics on early context cancellation.Known Limitations & Trade-offs
SFTP is not an object store, so mapping it to the
blobinterface requires a few compromises, which are documented in the package docstring:Pagination (
ListPaged): SFTP lacks native server-side pagination or cursors.ListPagedis implemented via physical directory walks. Users should be warned that this is O(N) in network calls and not recommended for directories with massive file counts.Server-Side Copying: Native server-side copying depends on the
copy-dataSFTP extension. If the target server (like older OpenSSH versions) does not support this,Copyfalls back toio.Copy, pulling the data to the client and pushing it back up.Metadata: Extended attributes (
CacheControl,ContentType, etc.) are stored in an adjacent.attrssidecar file via JSON. This can be disabled via?metadata=skipin the connection URL.Testing Strategy
drivertestconformance suite.Checklist
Added URL opener examples.
Ran
go fmtandgo vet.Verified
drivertest.RunConformanceTestspasses.