Emulate Common TLS Fingerprints + HTTP/SOCK5 Proxy Support

[Silvenga]

As noted here #446 (comment) - I suspect we are being blocked automatically by Fastly bot detection.

So as a fix for the current TLS block, this pr switches outgoing HTTP requests to use wreq - designed to emulate popular browsers fingerprints (TLS, JA3/JA4, and HTTP/2). I opted to using a small pool of popular browsers/os, one randomly used.

Existing hyper responses handing was preserved with a compatibility layer. This introduces using the BoringSSL TLS stack (built from source by rust, so it introduced some build complexities).

I also added proxy support since it was a wreq option (#458) and moved the tests to be consistently in a test mod.

TODO

• Fix the alpine build. @gmemstr
• Fix the dev container.
• Fix Nix. Does this work? Emulate Common TLS Fingerprints + HTTP/SOCK5 Proxy Support #544 (comment)
• Pull the client accept header in the proxy endpoints for resources.

[ButteredCats]

Successfully lets me fetch pages from Reddit where I'm usually blocked. However, all images meant to be proxied from Reddit except emojis are giving a 307 redirect to their Reddit counterpart which results in them not loading.

[jnobbe]

Pardon my ignorance, but does this need to be built or can it be referenced in a compose file? @Silvenga @ButteredCats

[Silvenga]

Give me a bit, I broke something in a recent commit. This would be building the Dockerfile.ubuntu.

[Silvenga]

Sorry about that, should be good now @ButteredCats @jnobbe

[ButteredCats]

All good lol. I'll apply the patch on one of my servers and let you know how it goes!

[jnobbe]

@Silvenga , this is my first time doing a build instead of pulling an image.

I was able to build using docker buildx build -f ./Dockerfile.ubuntu -t Silvenga/redlib .

I swapped the new image name into my compose and I'm now getting an error about an env not being found in the build directory. I don't know if this is due to a (likely) oversight on my part.

Also, in the readme I saw you added info about boringssl. Is that included as a part of what you've done, does it need to be done on the host, or something else?

[dave0003]

This is working for me so far (pr509 is still working for me, also).

[dave0003]

I was also able to use a socks proxy by adding something like this to my systemd service file under [service]:

Environment="https_proxy=socks5h://127.0.0.1:8430"

Very nice to have socks support, thank you.

[jnobbe]

OK, I found the solution to the env error and successfully pulled and built the PR. I'll report back any issues @Silvenga .

[ButteredCats]

Works for sure and I think this is the way to go! Still regularly getting ratelimited but I don't think there's a good fix for that.

[oifj34f34f]

Hi! I want to make some changes to redlib, so I cloned the repository (this fork) via VSCode (Dev Containers extension), but I'm getting:

0.299 (!) The 'moby' option is not supported on Debian 'trixie' because 'moby-cli' and related system packages have been removed from that distribution.
0.299 (!) To continue, either set the feature option '"moby": false' or use a different base image (for example: 'debian:bookworm' or 'ubuntu-24.04').
0.299 ERROR: Feature "Docker (Docker-in-Docker)" (ghcr.io/devcontainers/features/docker-in-docker) failed to install! Look at the documentation at https://github.com/devcontainers/features/tree/main/src/docker-in-docker for help troubleshooting this error.

What am I doing wrong? I'm using Docker (WSL2) on Windows 11.

Actually, disabling the moby option helps:

{
    "name": "Rust",
    "image": "mcr.microsoft.com/devcontainers/rust:dev-trixie",
    "features": {
        "ghcr.io/devcontainers/features/docker-in-docker:2": {
            "moby": false
        }
    },
    "portsAttributes": {
        "8080": {
            "label": "redlib",
            "onAutoForward": "notify"
        }
    },
    "postCreateCommand": "sudo apt-get update && sudo apt-get install -y git build-essential cmake libclang-dev",
    "postStartCommand": "cargo build"
}

But can we do this by default? Or is the issue on my end (even though I haven't changed anything)? What is the correct way?

[Silvenga]

Hi! I want to make some changes to redlib, so I cloned the repository (this fork) via VSCode (Dev Containers extension), but I'm getting:

TBH, I don't use dev-containers, so I'm not sure - I just kind of guessed at what the updates needed to be hoping someone that uses them would come by 😁. Regarding trixie, I just took the liberty in applying security updates, so I also updated the image OS's to latest LTS - it's not strictly needed, of course those other solutions also seem legit.

[erusc]

Just out of curiosity, wouldn't it appear suspicious to emulate Windows or regular browsers?
Because when visiting via a desktop browser, there's no request (as shown in DevTools) to OAuth endpoints like /auth/v2/oauth/access-token/loid.

[Silvenga]

@erusc so my logic is that it's Fasty just doing bot detection - and not someone actively trying to block Redlib.

I don't think it would be worth it to Reddit to have an engineer manually annoy Redlib users, it's just not cost effective. Engineering doesn't want to block us, leadership does. It would be vastly more cost effective to pay the engineers to build generic blocking solutions that automatically adapt - or even better, outsource this to their CDN (Fastly).

So let's focus on Fastly bot detection. Fastly is looking for bots - typically programs using the standard libraries without customization. Now, Fastly isn't going to just have engineers go out and find the FPs of hundreds of developer libraries - what they are going to look at are FPs that are both somewhat common, yet not incredibly common (like from legit browsers they manually test). It's "machine learning" - getting signals and running a statistical analysis for outliers and blocking based on what they believe is a reasonable risk (unlikely to block someone legitimate).

Fastly isn't going to look at the context of the request (a mobile request), it's just looking at the surface signals (is this request statistically likely to be from a bot?). Would it be suspicious to a human? Yep! To a generic statistical engine? Highly unlikely.

Ultimately, we want to blend into the crowd here. The OS really just modifies the user-agent string, iirc and sets the other common headers like sec-ch-ua. We are TLS FPs right now, the OS emulation is just part of the package that also future proofs us against future advancements in bot detection (e.g., looking at the headers, looking at the case of the headers, order of the headers, etc.).

I picked Windows because that's just the most popular client out there. Android was added to increase evasion while ideally having a high chance of different Redlib instances picking the same device/browser to emulate (for privacy of long running instances, which felt likely).

[nomadlogic]

I totally agree with your analysis @Silvenga - I'm running this PR on my FreeBSD based system now (just got blocked running the main redlib release) and its looking great so far. I just needed to make sure llvm and cmake were available, but other than that it just works. cheers!

[gmemstr]

Something in Dockerfile.alpine is very broken (bails with a permission denied error)

{"msg":"exec container process `/usr/local/bin/redlib`: Permission denied","level":"error","time":"2026-04-11T11:50:37.481935Z"}

Brute forcing it with a chmod 777 doesn't resolve it, but it can run as root. Which feels extra weird.

/ # ldd /usr/local/bin/redlib
        /lib/ld-musl-x86_64.so.1 (0x7fe336620000)
/ # ls -alh /lib/ld-musl-x86_64.so.1
-rwxr-x---    1 root     root      650.6K Oct 13 18:32 /lib/ld-musl-x86_64.so.1

[philpax]

For anyone who wants to use this fork as a drop-in replacement in NixOS:

{ config, lib, pkgs, unstable, ... }:

let
  src = pkgs.fetchFromGitHub {
    owner = "Silvenga";
    repo = "redlib";
    rev = "af002ab216d271890e715c2d3413f7193c07c640";
    hash = "sha256-Ny/pdBZFgUAV27e3wREPV8DUtP3XfMdlw0T01q4b70U=";
  };
  # Use Silvenga's wreq fork (redlib-org/redlib#544) which uses BoringSSL
  # to emulate browser TLS fingerprints and evade bot detection
  redlib-fork = unstable.redlib.overrideAttrs (oldAttrs: {
    version = "0.36.0-unstable-2026-04-04";
    inherit src;
    cargoDeps = unstable.rustPlatform.fetchCargoVendor {
      inherit src;
      name = "redlib-0.36.0-unstable-2026-04-04-vendor";
      hash = "sha256-eO3c7rlFna3DuO31etJ6S4c7NmcvgvIWZ1KVkNIuUqQ=";
    };
    # BoringSSL (via boring-sys2) needs cmake, go, git, perl, and libclang for bindgen
    nativeBuildInputs = (oldAttrs.nativeBuildInputs or []) ++ (with pkgs; [
      cmake
      go
      perl
      git
      rustPlatform.bindgenHook
    ]);
    checkFlags = (oldAttrs.checkFlags or []) ++ [
      "--skip=oauth::tests::test_generic_web_backend"
      "--skip=oauth::tests::test_mobile_spoof_backend"
    ];
  });
in
{
  services.redlib = {
    enable = true;
    package = redlib-fork;
  };
}

With that, I can confirm that this fork works for me - thank you very much 🙂

[sigaloid]

Overall great at first pass, I'm happy to move in this direction to be more in line with bypassing techniques as they implement blocks. Going to look more into it soon

[sigaloid]

Could we target a more specific subset, or even just the exact content-type ahead of time (from filename)? Or is an accept of */* not an uncommon thing?

[Silvenga]

Woot, you're alive!

Regarding this,

I added it because without a valid accept, Reddit tries redirect to a HTML page that just inlines the resource. I added it after removing this:

redlib/src/oauth.rs

Line 349 in ba98178

builder = builder.header("Accept", "*/*");

But now I'm looking and I'm not sure how the original code worked here (since I don't see where headers are set).

I don't think we should try and parse it from the URL, that would be against modern "spec" (this would be under "mime type sniffing" I think). Browsers figure out the Accept header from context e.g., a img tag means send all the image mime types. Since Redlib doesn't know the context of a resource, it would ideally treat the resource as opaque (hence Accept everything).

But now I'm wondering if the actual solution is to pass through the client's accept header... which I'm wondering if the original code actually did. I might have lost it with the Hyper request compatibility shim... that seems like the best solution, let the browser tell us what it thinks the resource should be...

[sigaloid]

😅 Life gets in the way. About to graduate! But luckily I dogfood redlib so significant breakage does still get me back here.

Yep, I remember the weird inlining behavior (at some point it was misconfigured so it would always return that page on browser which was really annoying).

I think that passing through the client's accept header could be a better solution as it would pass through from the browser's intentions. (Honestly don't think we did that before?...)

It relates to the general philosophy of either letting the client send things mostly unmodified (which can open up concerns about nel, report-to, etc being passed through to "smart" features that browsers implement), vs trying to be "smart" about what the client really wants (like reddit->redlib url rewriting; we decided that was a common enough use case to support, but maybe it's bad to do that - there's some use cases that could be valuable to have the option to report the raw data? like academic / archival - I've had to strip these features before for users like these)

I'd say for now we could go with accept */* - I'm open to hearing more on this, and may be worth adding this to the comments - since this is a change in behavior from main.

And I'm really glad you did this work especially to come off of hyper - there was a lot of technical cruft with the old stack that I could never prioritize cleaning out. :)

For now, I think we're at a stage that this can go in today/tomorrow when I get to once-over again, and we can iterate on main if there's any problems. Being subject to the whims of an ML bot detection framework means that's kind of our only choice is to experiment. The moment we migrate, their ML could notice the new traffic signature and pivot quickly (hi Fastly SWE!) based on load, so no amount of testing could really come to any conclusions early :)