VISUALPING Token Hunt Crawler

Two crawlers, both authenticating with HTTP Basic Auth and staying within the target's domain, breadth-first from a homepage. Both scan every body, header, cookie, and image-metadata field they see for the pattern VISUALPING\{[0-9a-f]{16}\} and log exactly where each match came from — the resource URL and the "channel" (body / header / cookie / EXIF field / ...) — so findings can be traced back to their source.

• crawler_BeautifulSoup.py (stage 1, static): requests + BeautifulSoup. Fast, but only sees what the server actually sent — no JavaScript execution.
• crawler_playwright.py (stage 2, rendered): a real headless browser (Playwright + Chromium). Slower, but sees what a browser actually renders — JS-injected links/content, console output, browser cookies, localStorage/sessionStorage, and every network request (XHR/fetch, images, scripts) fired while a page loads, not just the original HTML.

Setup

Copy the example env file and fill in the target and credentials:

cp .env.example .env

.env:

TARGET_URL=http://54.214.7.161/
CRAWL_USER=your-username
CRAWL_PASS=your-password

Run with Docker (recommended — sandboxed)

Build the image (this also installs Chromium for stage 2 and the Tesseract OCR engine, so the build takes a while the first time):

docker build -t visualping-crawler .

Run stage 1, mounting logs/ and resources/ so output lands on your host:

docker run --rm --env-file .env -v "$(pwd)/logs:/app/logs" -v "$(pwd)/resources:/app/resources" visualping-crawler

Run stage 2 by overriding the command:

docker run --rm --env-file .env -v "$(pwd)/logs:/app/logs" visualping-crawler python crawler_playwright.py

On Windows PowerShell, replace $(pwd) with ${PWD}.

Run locally (without Docker)

Requires Python 3.10+.

python -m venv venv
source venv/bin/activate   # venv\Scripts\activate on Windows
pip install -r requirements.txt
playwright install chromium   # only needed for stage 2
python crawler_BeautifulSoup.py   # stage 1
python crawler_playwright.py      # stage 2

OCR (image-text scanning) also needs the Tesseract binary installed separately — it's not pip-installable. On Debian/Ubuntu: apt-get install tesseract-ocr; on Windows, the UB-Mannheim installer. If it isn't found, both scripts log one warning and keep running without OCR rather than crashing — Docker is the simplest way to get a guaranteed-working setup.

Output

Each run creates a timestamped log file under logs/ (stage 2's are prefixed dynamic_), echoed to stdout too. Fetch lines look like:

200  text/html                       http://54.214.7.161/            (1234 bytes)
200  text/css                       http://54.214.7.161/style.css    (512 bytes)
404  text/html                       http://54.214.7.161/missing     (301 bytes)

Resource-specific lines:

METADATA  http://.../logo.png    {"format": "PNG", "size": [64, 64], "info": {"Comment": "..."}}
STORED    http://.../app.js      ->  resources/3f9a1c2b7e4d5f60.js

Text-based resources (JS, CSS, JSON, XML, SVG, plain text) are saved under resources/, named by a hash of their URL — the STORED log line maps back to the source URL.

Token matches:

FOUND  (3/8)  VISUALPING{...}  <-  http://.../analytics.js [body]
MATCH  (seen before)  VISUALPING{...}  <-  http://.../other-page [header:x-provisioning-note]

FOUND marks the first time a token is seen; MATCH logs every additional place the same token turns up (useful for a write-up — the same token sometimes appears via more than one channel). Channels seen so far: body, header:<name>, cookie:<name>, exif:<field>, info:<field>, ocr, and — stage 2 only — rendered-dom, network-body, console, localStorage:<key>, sessionStorage:<key>. Each run ends with a summary listing every token found and all of its locations.

Any of the above channels can also carry a :charcode-array suffix (e.g. body:charcode-array) when the token wasn't literal text but was reconstructed from a JS array of character codes (String.fromCharCode.apply(null, [86, 73, ...])) — a common way to keep the literal string out of the source. Similarly, ocr:corrected means the token was read from an image via OCR and had to be fixed up first, because Tesseract commonly misreads 1 as l and 0 as O/o — worth a quick visual double-check of the source image if you're citing it in a write-up.

Notes / current scope

Stage 1 (crawler_BeautifulSoup.py)

• Follows <a href="..."> links for the page crawl; also fetches <img src>, <script src>, <link href>, <embed src>, <object data>, <iframe src>, and URLs found via regex inside inline <script>/<style> text and style="" attributes, as resources.
• If a resource turns out to actually be HTML (e.g. an <iframe> embedding another page), it's parsed for further links/resources too.
• A resource that is CSS or JS gets one extra hop of the same url(...)/string-literal regex applied to its body (catches things like a stylesheet's url(font.woff)).
• Does not execute JavaScript, so JS-injected links/content are not seen here — that's what stage 2 is for.
• Every image is OCR'd (in addition to EXIF/metadata), for tokens drawn directly into the pixels rather than stored as text anywhere.

Stage 2 (crawler_playwright.py)

• Re-parses the rendered DOM (page.content() after JS has run) for links, so anchors injected by JavaScript get followed too — not just what was in the original server response.
• Every network response Chromium fires while rendering a page (the page itself, images, scripts, XHR/fetch) is logged and scanned, which is how it picks up content a plain GET never triggers.
• A renderer crash (Chromium's "Page crashed", seen once against a heavy page on this target — possibly a deliberate trap for browser-based crawlers) is caught, and the whole browser session is relaunched so one bad page doesn't end the crawl; the crashing URL itself is simply skipped, not retried.
• Does not click buttons or elements without an href/src — it discovers what JS injects into the DOM/network automatically, but won't trigger interaction-only content (e.g. a menu that only fetches data on click).

Shared

• Both stay within the starting page's domain (scheme://host:port).
• Both track visited URLs (post-redirect too) to avoid re-fetching and loops.
• MAX_PAGES (default 1000) caps the total crawl as a safety net.
• MAX_PER_PATH (default 15) caps how many times a given path is fetched regardless of query string, to survive infinite-pagination traps (/report/?page=N for any N) and decoy tracking params (?ref=, ?utm_source=, ...) that otherwise look like endless "new" URLs. Set to 0 to disable. Skipped fetches are logged as SKIPPED (path cap N reached).