Origin Discovery (CDN/WAF unmasking)

The Origin Discovery module finds the real origin server hiding behind a CDN or WAF (Cloudflare, Akamai, Fastly, CloudFront, Imperva, Sucuri, and friends), and proves it is reachable directly, without the shield.

It runs as a GROUP 6 Phase A scanner (parallel with Nuclei, GraphQL Security, Subdomain Takeover, VHost & SNI, and Web Cache Poisoning). It is disabled by default. Enable it in the project settings under the Discovery & OSINT tab, or run it on demand from the workflow graph (see Partial recon).

Badge: both. It mixes passive source queries (public catalogues and DNS) with a small amount of active validation traffic to the candidate servers.

The core concept

Think of a company that never publishes its office address. All its mail goes to a PO box, and the post office quietly forwards it to the real building. The PO box is the CDN/WAF. The real building is the origin server. When anyone looks up the site, they only ever see the PO box; every protection (the firewall, the bot filtering, the DDoS defence, the rate limiting) lives at the PO box.

If an attacker learns the real building's address and knocks on its door directly, they walk straight past every one of those protections. The origin usually serves the same application as the fronted site, but with none of the edge controls in front of it. So the whole game is: find the hidden building.

You can never simply ask for the hidden address. But the origin keeps leaving the same fingerprints in public places, and there are giant public catalogues of "every server on the internet and what it looks like" (Shodan, Censys, FOFA, ZoomEye) plus historical records of where names used to point (passive DNS, DNS history, certificate transparency). Origin Discovery pivots on each fingerprint to gather candidate origin IPs, then confirms each one by knocking on its door and comparing what comes back to the real site.

Why the origin leaks (the fingerprints)

Every discovery source exploits one way the origin gives itself away:

None of these proves anything on its own; each yields suspects. Confirmation is a separate, deliberate step (the validator).

Pipeline position

GROUP 4  HTTP probing (httpx)  -> per host: is_cdn / cdn name / favicon_hash / server / status
GROUP 5  Resource enumeration
GROUP 6 Phase A  (parallel fan-out)
         Nuclei || GraphQL || Subdomain Takeover || VHost & SNI || Web Cache Poisoning || ORIGIN DISCOVERY

Origin Discovery is a post-HTTP-probe active scanner. It needs two things httpx produces in GROUP 4:

1. the CDN classification (is_cdn, the CDN name) so it knows which hosts are fronted and worth unmasking, and
2. the favicon hash for the Shodan/FOFA/ZoomEye favicon pivots.

Each Phase A tool runs on its own deep-copied snapshot of the shared recon result (run_origin_discovery_enrichment_isolated), so the parallel threads never race on the same dict. In IP-mode scans the security-checks producer runs sequentially before Phase A, so Origin Discovery and check_waf_bypass never overlap; in domain-mode they run as sibling Phase A threads and their graph writes are serialised through a single-worker executor.

How a scan flows (the macro view)

1. SELECT fronted hosts   from http_probe.by_url: is_cdn flag, or the edge IP sits in a known CDN range
       |
2. GATHER candidates      per fronted host, from every enabled source (never-raise, per-target cache):
       keyless : non-CDN subdomains ; SPF/MX email IPs ; crt.sh SANs ; favicon mmh3
       scanners: Shodan (favicon + cert-CN + hostname) ; Censys/FOFA/ZoomEye (cert + favicon) ; OTX/VT passive DNS
       passive : SecurityTrails ; ViewDNS  (DNS history)
       |
3. FILTER (fail-closed)    drop non-routable/internal/metadata IPs (SSRF gate) ; drop CDN-range IPs ;
                           drop the host's CURRENT DNS resolution (that's the edge) ; drop RoE-excluded ;
                           cap at ORIGIN_DISCOVERY_MAX_CANDIDATES
       |
4. CONFIRM                 fetch the fronted host (reference) + each candidate IP directly and via a forged
                           Host header ; score HTML 60% + cert 25% + headers 15% (± status) ; confirm > threshold
       |
5. WRITE                   IP node (+ origin provenance) ; Subdomain-[:HAS_ORIGIN]->IP ; Vulnerability(waf_bypass)

Discovery sources

Sources are organised into three groups, each independently toggleable. A source with no key configured is skipped silently; a source that errors records a soft marker and logs [-][OriginDiscovery] host: source 'x' failed …; it never sinks the run (the never-raise contract).

Keyless group (originDiscoveryKeyless, default on)

Works with zero API keys; this is what keeps the module useful out of the box.

• Non-CDN subdomain probing. Resolves 44 origin-leaking subdomain names against the target domain (mail, webmail, smtp, pop, imap, ftp, sftp, cpanel, whm, plesk, webmin, direct, origin, origin-www, direct-connect, dev, staging, stage, test, qa, uat, api, backend, admin, panel, old, legacy, backup, bak, ns1, ns2, dns, vpn, remote, gateway, mx, mx1, mx2, mailgw, autodiscover, autoconfig, portal, intranet, internal). Any that resolve to a routable, non-CDN IP become candidates. This is the highest-yield keyless technique.
• SPF / MX email records. Parses the domain's TXT SPF record for ip4: / ip6: entries (expanding CIDRs up to /24), and resolves MX hosts to IPs, skipping managed mail vendors (google, outlook, microsoft, mimecast, proofpoint, barracuda, pphosted) whose IPs are never the origin.
• crt.sh certificate transparency. Queries crt.sh for every certificate naming the domain, extracts the SAN hostnames, and resolves them to off-CDN IPs (reuses RedAmon's query_crtsh).
• Favicon hash. Read from the httpx probe when available; otherwise the module self-fetches /favicon.ico (SSRF-guarded, rate-limited) and computes the hash itself. Feeds the scanner favicon pivots below.

Scanner group (originDiscoveryScanners, default on)

Reuses the internet-wide scanner keys you already store in Global Settings. Each source is skipped if its key is absent.

Key delivery (important). These keys are normally only loaded into the scan when their own enrichment tool is enabled. Origin Discovery widens that gate: with Origin Discovery + its scanner group on, the stored Shodan/Censys/FOFA/ZoomEye/OTX/VT keys are delivered even if those tools' own toggles are off, so the favicon/cert pivots actually fire. (Without this widening they would silently no-op; internally tracked as fix G10.)

Passive-DNS group (originDiscoveryPassiveDns, default on)

Historical A records that reveal the pre-CDN origin. These use two net-new keys.

The favicon hash

The favicon fingerprint is the Shodan/FOFA/ZoomEye standard: MurmurHash3 over the MIME-wrapped base64 of the raw icon bytes (base64 with a newline every 76 characters), cast to a signed 32-bit int. RedAmon's httpx probe already computes this exact value and stores it per host, so the module reuses it directly; it only recomputes (self-fetching /favicon.ico) when httpx did not run the favicon probe (for example under stealth mode, which disables it). The self-fetch is an extra active request to the target, so it is routed through the module's rate limiter and only performed when the host passes the SSRF safety check.

Candidate filtering (fail-closed)

Every candidate IP comes from untrusted third-party data (SPF ranges, Shodan/Censys results, passive DNS) or from the target itself, so each is filtered before it is ever probed or recorded. This is the module's most important safety control. A candidate is dropped, and counted under candidates_meta[host].dropped, if it is:

1. Non-routable / internal / metadata (the SSRF gate). is_non_routable_ip rejects RFC-1918 private, loopback, link-local, CGNAT (100.64.0.0/10), IETF-reserved, multicast and unspecified addresses (including 169.254.169.254 and IPv4-mapped IPv6 forms), and fails closed on anything it cannot parse. Without this, the weighted validator (which blindly connects to https://<ip>) would be an SSRF pivot into the internal network or cloud metadata.
2. A CDN/WAF edge IP. Checked against a ported list of Cloudflare / Akamai / Fastly / CloudFront / Imperva / Sucuri ranges and against the CDN flags already in the recon result: the edge is not the origin.
3. The host's current DNS resolution. The address the name resolves to today is the edge, not a find.
4. On the engagement's excluded-hosts list. Honoured via the shared exclusion filter.

Whatever survives is capped at ORIGIN_DISCOVERY_MAX_CANDIDATES (default 25) per fronted host.

The weighted-similarity validator

Confirmation reproduces unwaf's model. For each surviving candidate the module first does a fast TCP connect to each of the web ports [80, 443, 8080, 8443, 8000, 8008, 8888, 9443] and skips any that are closed or filtered (so a firewalled port never costs a full HTTP + TLS timeout). For each open port it fetches the candidate two ways (a direct request, and a request carrying a forged Host: header for the fronted domain) and scores each response against the reference (the fronted host fetched once up front):

overall = 0.60 · HTML_similarity
        + 0.25 · TLS_cert_match      (0.50 serial + 0.25 CN + 0.25 SAN overlap)
        + 0.15 · header_match        (Server + X-Powered-By exact, Set-Cookie name overlap)
        ± status_code_adjustment     (equal & ≥400: −0.10 ; equal & OK: +0.05 ; OK/error mismatch: −0.20)

• HTML similarity is a difflib ratio over the extracted text of each page (tags stripped), capped at 64 KB per side to bound cost.
• The result is clamped to [0, 1], multiplied by 100, and the candidate is confirmed when the score exceeds ORIGIN_DISCOVERY_THRESHOLD (default 60).
• A response is discarded before scoring if it is a 5xx or still carries WAF/CDN response headers (cf-ray, x-amz-cf-id, x-sucuri-id, a cloudflare/akamai/… Server value, …), which means the request went through the edge, so it is not a bypass.
• Empty-body / API fronted hosts: when the fronted host serves no comparable HTML (a JSON API, a 204, an empty redirect target), the 60% HTML weight is unavailable and the score can never reach the threshold. In that case an exact TLS match (cert component ≥ 0.5: a serial match, or matching CN plus all SANs) is treated as a definitive same-server signal and confirms the origin on its own. Without this, origins behind non-HTML front-ends would be silently missed.

Origin Discovery's validator (_score_candidate) is a standalone scorer: it does not reuse the security-checks check_waf_bypass routine, and it carries no AI/LLM classifier step (internally G4), so a scan never fires an LLM call per candidate; confirmation is purely the weighted HTML/cert/header score above. It still converges in the graph with any check_waf_bypass finding for the same exposure; see Deduplication and tenant isolation.

Search budget and per-target cache

• Per-target cache. Every per-domain search (favicon hash, cert-CN, passive DNS, crt.sh, SPF/MX, subdomain probing) is memoised for the run by (source, domain/host). So N fronted subdomains of one domain hit each API (and each 500-name crt.sh resolution) once, not N times. Only non-empty results are cached, so a transient failure on one host does not blank the source for the rest.
• Search budget. A per-scan ORIGIN_DISCOVERY_MAX_SEARCH_CALLS budget (default 50) is decremented only on a genuine, non-cached keyed scanner search. When it reaches zero the remaining keyed sources are skipped and a soft marker is recorded; this stops Origin Discovery from starving the sibling Shodan/Censys modules of query credits. Cached hits never draw the budget down.

Concurrency, rate, and safety model

• Concurrency. Fronted hosts are processed one at a time; within a host, candidate validation fans out through a bounded ThreadPoolExecutor sized by ORIGIN_DISCOVERY_WORKERS (default 10).
• Rate. All active traffic (validation probes, subdomain resolution, the favicon self-fetch) passes through a per-module rate limiter. Under Rules of Engagement with a global RPS cap, the module's rate key is capped alongside Naabu/httpx/Nuclei by the same governor. Under stealth mode the workers drop to 1, the rate to ~1 rps, and the loud keyed scanner searches are turned off.
• Trust boundaries. The dangerous crossing is recon container → candidate IP: target- and third-party-controlled data becomes a probe destination. That crossing is gated by the fail-closed SSRF filter above. Candidate probes never follow redirects (a 3xx could point at an internal host the IP filter never saw); only the in-scope reference fetch follows its canonical redirects. IPv6 candidates are bracketed so their URLs are well-formed. Third-party responses are treated as untrusted: bodies are byte-capped, cert SANs are stored via parameterised Cypher (never interpolated), and no untrusted content reaches an LLM prompt.
• Secrets. The SecurityTrails and ViewDNS keys live in per-user Global Settings, ride the existing config path into the scan, and are never logged or sent to the frontend; the ViewDNS URL (which embeds the key) is never written to a log line.

Full parameter reference

Set these in Discovery & OSINT → Origin Discovery. The name triad is Prisma originDiscovery* / Python ORIGIN_DISCOVERY_*.

Master toggle & source groups

Tuning

An internal ORIGIN_DISCOVERY_RATE (rps ceiling, 0 = unlimited) is not surfaced in the form; it exists so the engagement rate ceiling can cap the module's active traffic. Note that 0 here means UNLIMITED, which is why the ceiling rewrites it downward rather than leaving it alone.

Keys (Global Settings)

securitytrailsApiKey, viewdnsApiKey are net-new. Shodan/Censys/FOFA/ZoomEye/OTX/VirusTotal keys are the ones you already configure for those enrichment tools and are reused here.

Output structure

The module writes one entry to the recon result:

"origin_discovery": {
  "confirmed": [
    {
      "type": "waf_bypass",
      "severity": "high",
      "name": "Origin Server Exposed (CDN Bypass)",
      "subdomain": "www.example.com",
      "matched_ip": "203.0.113.42",
      "url": "https://203.0.113.42:443",
      "origin_discovery_method": "favicon_hash",   // subdomain | email_record | cert_san | favicon_hash | passive_dns
      "origin_source": "shodan",                    // shodan | censys | fofa | zoomeye | otx | virustotal | securitytrails | viewdns | crtsh | dns
      "confidence_score": 91.0,
      "cdn_fronting": "cloudflare",
      "port": 443,
      "match_method": "host-header",                // direct | host-header
      "html_similarity": 94.2,
      "cert_match": 100.0,
      "header_match": 66.7,
      "status_code": 200,
      "evidence": "www.example.com → 203.0.113.42:443 (shodan, host-header, score 91.0%)",
      "source": "origin_discovery"
    }
  ],
  "candidates_meta": {
    "www.example.com": {
      "candidates": ["203.0.113.42", "..."],
      "methods": ["favicon_hash", "cert_san", "passive_dns"],
      "dropped": { "ssrf": 3, "cdn": 5, "current_resolution": 1, "roe": 0 },
      "errors": { "shodan": "invalid key (401)" }
    }
  }
}

candidates_meta is your audit trail: which sources ran, how many candidates each drop-reason removed, and which sources failed and why.

Graph schema

Origin Discovery reuses the existing IP and Vulnerability labels (no new node type) plus one new relationship, HAS_ORIGIN. Writer: graph_db/mixins/osint_mixin.py::update_graph_from_origin_discovery. See also Attack Surface Graph.

Consumed (input)

• Subdomain / IP: CDN-fronted hosts carrying an is_cdn flag or a favicon hash from a prior HTTP probe.

Produced / enriched

IP node (tenant-keyed {address, user_id, project_id}) gains:

Vulnerability node: type: "waf_bypass", source: "origin_discovery", with matched_ip, hostname, canonical url/matched_at = https://<ip>, the specific probe_url, port, confidence_score, html_similarity/cert_match/header_match, status_code, cdn_fronting, evidence.

Relationships

• (:Subdomain)-[:HAS_ORIGIN {method, confidence, origin_source, discovered_at}]->(:IP)
• (:IP)-[:HAS_VULNERABILITY]->(:Vulnerability)
• (:Subdomain)-[:WAF_BYPASS_VIA {evidence, discovered_at}]->(:IP), the existing WAF-bypass edge.

Deduplication and tenant isolation

The Vulnerability id is a deterministic, tenant-scoped hash of (type, url, ip, user_id, project_id). Consequences:

• Idempotent: re-running a scan MERGEs onto the same node, never duplicates.
• Converges with the security-check producer: an origin exposure that check_waf_bypass also finds for the same (type, https://ip) lands on one Vulnerability node in the same project, so reports do not double-count it.
• Never leaks across tenants: two projects that discover the same origin IP get separate nodes (the tenant is in the hash), so one project's finding can never overwrite another's.

A project wipe (clear_project_data) removes the origin nodes and the HAS_ORIGIN edge with no per-type code, because every node carries the tenant key.

Engagement limits

The limits are enforced in code, not just prose. The excluded-hosts list drops any candidate IP or subdomain that matches it, and the global RPS cap governs the module's active probes alongside every other active tool.

The MECHANISM is unchanged; what changed is where the fields live and who may write them. roeGlobalMaxRps and roeExcludedHosts are ordinary project settings now, in Target & Modules, editable at any time and writable over the API like any other field. They are not a separate block with its own rules. What makes that safe is the same thing that always made the cap real: it is applied at scan start to every rate field, including ORIGIN_DISCOVERY_RATE, whatever the per-tool value says.

See Rules of Engagement.

Recon presets

Origin Discovery is enabled in intent-appropriate presets: bug-bounty-deep, cloud-exposure, red-team-operator, and infrastructure-mapper. It is explicitly off in full-passive-scan and stealth-recon, which avoid the active validation probing (stealth-recon can still be customised to run it throttled; see below).

Stealth mode

Under stealth mode the module stays enabled (unmasking is the point of a stealth engagement) but its active footprint is clamped hard: ORIGIN_DISCOVERY_WORKERS = 1, ORIGIN_DISCOVERY_RATE = 1 rps, and the keyed scanner searches (originDiscoveryScanners) are turned off. The keyless and passive-DNS sources (which do not send loud direct-IP traffic) stay on.

Partial recon

Origin Discovery can be run on its own from the workflow graph without a full scan. See Recon Pipeline Workflow: Partial Recon.

• Inputs. It reads the CDN-fronted hosts already in the graph: a Subdomain or IP that a prior HTTP probe classified as fronted (the Endpoint carries is_cdn or a favicon hash, or the host resolves to a CDN IP). If the project has no such hosts and you enter no subdomain, the modal blocks the run with a hint to run an HTTP-probe scan first (there is nothing to unmask).
• Manual entry. You may type a Subdomain to unmask (it is marked as fronted and processed). IP is graph-sourced only: a discovered origin IP is an output of this tool, never a manual input, so the modal offers no IP field.
• Scope. The apex-domain toggle (Include Root Domain) is honoured identically to the full pipeline, so a partial run never writes apex nodes the engagement's scope excludes.

Reporting

A confirmed origin flows into the pentest report through the existing WAF-bypass path (it is a Vulnerability of type waf_bypass, severity high, attached to the origin IP) and contributes to the risk score like any other high finding. Because the id converges with the security-check producer, the same exposure is counted once.

Known limitations (be honest)

• HTML-similarity relies on a comparable page. The primary signal is the page body. A fronted host that serves an empty or non-HTML front-end is confirmed only via an exact TLS match; a host with neither comparable HTML nor a matching certificate can be missed.
• Needs a prior HTTP probe. The module unmasks hosts that GROUP 4 already classified as CDN-fronted. A brand-new project with no HTTP-probe data has nothing to work on (partial recon states this explicitly).
• Registrable-domain heuristic. The domain used for SPF/crt.sh/passive-DNS queries is derived with a curated public-suffix list covering the common multi-label ccTLDs (.co.uk, .com.au, .co.jp, …). A very obscure suffix outside that list falls back to a two-label guess.
• Keyless-only coverage is narrower. With no scanner or passive-DNS keys, discovery leans on subdomain/SPF/crt.sh; the favicon and cert-search catalogues (the strongest pivots) need keys.
• /24 neighbour expansion is out of scope for this module (it belongs to a separate ASN/CIDR-expansion feature).

Reference and provenance

The mechanism is a faithful Python re-implementation of unwaf (Go), studied at commit c8302eb. RedAmon does not ship or run the unwaf binary and vendors none of its Go code; it reproduces the behaviour (the source set, the exact scoring weights and default threshold, and the favicon hashing) in Python, adding RedAmon's SSRF/RoE/budget/AI-off guards on top. The reason for re-implementing rather than wrapping the binary is control of the active probing: every connection to a candidate must sit behind RedAmon's own SSRF filter and RoE scope, which a sealed binary cannot guarantee.

Test coverage

• recon/tests/test_origin_discovery.py: source parsing, the SSRF/RoE/CDN/current-resolution filter, weighted-scoring confirm/reject, the WAF-header rejection, the empty-reference cert path, IPv6 bracketing, the per-domain cache, budget, never-raise, and the isolated-wrapper no-mutate contract.
• recon/tests/test_origin_discovery_settings.py: the settings cascade, the reused-key delivery gate, and the stealth-mode throttle.
• recon/tests/test_partial_recon.py::TestRunOriginDiscovery: the partial-recon entry, force-enable, graph-fronted injection, and IP-input rejection.
• tests/test_origin_discovery_graph_live.py: a live-Neo4j proof of the graph write: node creation, idempotent re-run, cross-tenant isolation, convergence with the security-check producer, orphan-free project wipe, and the fronted-host count.

Related pages

• VHost & SNI Enumeration: the sibling Phase A active scanner.
• Rules of Engagement · Recon Presets · Recon Pipeline Workflow: Partial Recon
• Attack Surface Graph · Pentest Reports