Gowitness fails to correlate screenshots/network logs/technologies due to URL mismatch

[aconite33]

Describe the bug
Gowitness crashes with KeyError (or after PR #2974, silently drops results with warnings) when trying to correlate screenshots, network logs, and technologies back to parent events. The root cause is that gowitness records URLs in its SQLite DB differently than the input URLs — it may change the scheme (e.g. https://host/ becomes http://host:443/) and/or change the port (e.g. http://host/ becomes http://host:443/ after a redirect to HTTPS). The event_dict lookup then fails because the keys don't match.

This affects screenshots, network logs, and technology correlation — so gowitness runs but produces no output for affected URLs.

BBOT Command

bbot -p preset.yml -m httpx gowitness -t targets.txt

OS, BBOT Installation Method + Version
Ubuntu, uv install, 3.0 branch

BBOT Config
Default gowitness config, targets include domains hosted behind CDNs that redirect HTTP to HTTPS.

Logs/Screenshots

Before PR #2974, gowitness crashes the batch entirely:

[ERRR] Error in gowitness.handle_batch(URL: 6): /root/bbot/bbot/modules/gowitness.py:184:handle_batch(): 'http://res.ml.tata.com:443/'
  File "/root/bbot/bbot/modules/gowitness.py", line 184, in handle_batch
    parent_event = event_dict[url]
                   ~~~~~~~~~~^^^^^
KeyError: 'http://res.ml.tata.com:443/'

After PR #2974 (which fixed network_logs and technologies but missed the screenshot section), gowitness still crashes on the screenshot lookup at line 184. After applying .get() to the screenshot section, the crash is gone but results are silently dropped:

[WARN] gowitness: Could not correlate screenshot to parent event for URL: http://m.ml.tata.com:443/
[WARN] gowitness: Could not correlate screenshot to parent event for URL: http://t.ml.tata.com:443/
[WARN] gowitness: Could not correlate network log to parent event for URL: http://m.ml.tata.com:443/
[WARN] gowitness: Could not correlate technology to parent event for URL: http://t.ml.tata.com:443/

Debug logs show the issue clearly:

• Input URL (from httpx): https://m.ml.tata.com/ or http://url6586.tata.com/
• Gowitness DB records: http://m.ml.tata.com:443/ or http://url6586.tata.com:443/

The scheme and/or port differ, so the event_dict lookup fails.

[liquidsec]

fixed in #2985