Inside the Recon Pipeline

Group 1 runs up to three jobs (WHOIS, discovery, URLScan) on a fixed pool of three, two in exact mode, where discovery never runs. The other pools are sized from the switches: group 3 has one worker for Shodan plus one per enabled port scanner; the OSINT group min(sources, 5); Phase A of group 6 one per enabled active scanner. The results are merged back on the main thread, one tool at a time, so no two tools write the same key at once.

Some steps run alone on purpose. Nmap needs the merged open ports of every port scanner. The TLS grab is sequential because the hostnames it adds change the target list the HTTP probe reads next. In Phase A each scanner works on its own deep copy of the results, so concurrent scanners cannot race on shared data. MITRE enrichment waits for Phase A because it enriches the CVEs Phase A found.

A group lasts as long as its slowest tool when every tool has its own worker; an OSINT group with more than five sources, and group 5b, take longer. Inside a tool there is more parallelism of its own: httpx threads, naabu's packet rate, Katana's parallel crawls. Those are tool settings, not pipeline groups.

A wildcard such as *.dev.acme.test is a group of its own, and exact hosts under it join it as seeds. Every other host joins the group of its registrable domain, looked up in the Public Suffix List, never by "the last two labels": shop.acme.co.uk belongs to acme.co.uk. A wildcard on a public suffix such as *.co.uk is refused, because it would enumerate everyone registered under it. The web app derives the groups for its preview, and the recon container re-validates them against its own copy of the list before it scans anything.

IP mode: from a range to a target list

The project form accepts single addresses and IPv4 ranges from /24 to /32 (at most 256 addresses). Each range is expanded to its usable host addresses (a /32 is the address itself), duplicates are dropped and the Rules of Engagement exclusions are applied. Reverse lookups run in parallel, up to DNS_MAX_WORKERS at a time (80 by default). WHOIS is asked once per /24 block, for the first address seen in it, rather than once per address.

Excluded hosts are applied where a host enters the run: a scan root on the list is refused before any tool or lookup touches it (even the ownership lookup), although the project's previous recon assets have already been cleared by then, and the list filters the hosts you typed, the subdomains discovery returns, the hostnames found in certificates and the addresses of an IP-mode range. An entry can be an exact host or IP, a domain (which also excludes its subdomains), *.domain (its subdomains only) or a CIDR range. The project form accepts exact hosts, domains, IPs and ranges from /24 to /32; *.domain entries and wider ranges come from the RoE document import or MCP. In IP mode an IP or CIDR entry removes addresses from the range. In domain mode it does not stop the port scan, the HTTP probe or Nuclei from probing an in-scope name that resolves into the excluded range; the TLS grab, the port security checks and origin discovery do skip those addresses.

The time window is checked by the orchestrator before it starts the container: outside the window the start is refused (403) and nothing runs. The container checks it again, once per domain run, before any target is touched; IP mode relies on the orchestrator's check alone. A run that starts inside the window runs to completion (only AI Surface Recon checks the window again and skips itself outside it); checking again per group would abandon a batch halfway and leave the graph half rebuilt. The rate ceiling lowers every rate setting the registry marks as capped (naabu, masscan, puredns, takeover and origin discovery among them) to the ceiling, including those whose 0 means unlimited. One exception today: stealth mode is applied after the ceiling and sets some of the same rates to its own values, so under a ceiling below them (1 request per second, say) Naabu still runs at 10. The ceiling is also per process, so a tool that runs several processes at once multiplies it.

Ownership verification looks for a TXT record at _redamon-verify.<scan root>, then at the registrable domain if it differs, holding redamon-verify=<token>. A resolver failure is not read as "no record": the group fails closed with the resolver's message, and the answer is not cached.

Where the session may go. By default, exactly where the scan goes: for a group that enumerates a root, its subdomains (*.root); for an exact group, the hosts you listed; the root itself only when Include root is on; in IP mode, the target addresses. Hosts your Rules of Engagement exclude are taken out. The profile's own scope hosts can only narrow this, never widen it: the session goes where both agree.

All or nothing. Most tools send one set of headers for their whole target list, so the session is attached only when every host on that list is in scope. One third-party host on the list and the tool runs logged out, with a log line saying so. The crawlers and fuzzers (Katana, Hakrawler, FFuf, ZAP, Arjun and Kiterunner) share one such check over all the hosts they crawl. httpx and Nuclei check their own lists and, when the session is attached, follow redirects only to the same host; Nuclei also drops its out-of-band callbacks, which go to a public third-party server. OpenAPI, AI Surface Recon and the cache-deception test decide host by host instead.

Values are checked, not cleaned. A recorded value comes from the target (it set the cookie), so a value with a line break or another control character, or longer than 8,192 characters, is refused outright: a silently trimmed cookie would be a different, broken one.

Five stages are gated by the scan modules list (domain discovery, port scan, HTTP probe, resource enumeration, vulnerability scan); every other stage has its own switch (MITRE also needs the vulnerability scan module, and Nmap and the TLS grab need port-scan data). In domain mode every stage also needs domain discovery in that list (it is there by default): without it the run loads the previous result file and runs only a short legacy sequence. One rule ties the web half to the network half: resource enumeration and the active scanners are skipped only when the HTTP probe ran and got no answer from any host (server errors still count as an answer). If the HTTP probe is switched off, they still run, seeded with http:// and https:// URLs built from the resolved hostnames.

Discovered subdomains. At the group-1 fan-in, a name that discovery volunteered is dropped if it resolves to loopback, link-local, multicast or the unspecified address. localhost.<domain> → 127.0.0.1 is a real, common record, and an address such as 169.254.169.254 is a cloud metadata service. No later stage checks this again, so this is the last chance. Hosts you typed yourself are never filtered: scanning them is your decision.

Certificate names. The TLS grab (group 3.6) and the HTTP probe's certificates (fed back right after group 4) can each add hostnames. So can Uncover's search results and the subdomains JS Recon finds in scripts. All of them go through the same filter, in this order:

• 1 · Under the root. A name that is not the scan root or below it becomes an external domain: recorded, never scanned. With no root to test against nothing is added at all; it fails closed. (IP mode never offers certificate names in the first place.)
• 1b · The group's own rule. An exact group adds nothing, because it scans only the hosts you listed. An enumerate group accepts strict subdomains, and the root itself only when the root is a target.
• 2 · Rules of Engagement. When Rules of Engagement are on, excluded hosts are removed.
• 3 · Syntax. Each label must be a valid hostname label. The check is anchored so that a newline hidden in a certificate name cannot split one target into two in the newline-separated target files.
• 4 · Cap. The certificate paths and JS Recon keep at most TLSX_MAX_INJECTED_HOSTNAMES names (200), sorted so the cut is reproducible; Uncover's names are not capped. The cap comes before the DNS lookups, so a certificate with thousands of names cannot stall the run.
• 5 · Resolve and check. A name that resolves to any non-routable address (private, loopback, link-local, reserved, multicast, CGNAT) is dropped. A name that does not resolve at all is kept: it is in scope, it cannot be an internal pivot, and a dead name in a certificate is a takeover lead.

Which tools touch private addresses

RedAmon scans internal networks on purpose, so a private address is not a reason to stop. But not every tool follows the same rule. Four guards exist: names discovered along the way may never point at loopback, link-local (cloud metadata lives there), unspecified or multicast addresses; anything sent to an outside service, and names injected from certificates, Uncover or JavaScript, must be publicly routable; links found in a target's JavaScript are fetched only if every address they resolve to is public; OpenAPI documents follow the first rule, except for hosts you typed. The port scan, the HTTP probe, Nuclei and the built-in checks have no address rule at all: the scope decides.

The built-in checks. Next to Nuclei, the vulnerability scan runs RedAmon's own checks: 33 you can switch off one by one, plus one that is always on, all on by default. They cover addresses that answer directly, bypassing the CDN; certificates and TLS; missing security headers; logins and session cookies sent without TLS; the organisation's DNS (SPF, DMARC, DNSSEC, zone transfer, once per organisation); exposed admin, database, Redis, Kubernetes and SMTP-relay services; unsafe page content; and missing rate limiting on logins. A few send real traffic of their own: a zone-transfer request, an SMTP relay attempt, a burst of 10 login POSTs. Stealth mode switches off ten of them.

The catch-all control. Many servers answer every path alike: an app shell returns its index page, an API router returns 200 {"code":"NOT_FOUND"}, an auth wall redirects everything to the login. When a check gets a hit, it asks the same server for a path that cannot exist, next to the hit, and compares the two answers. If they are the same page, the hit means nothing. The API-on-a-bare-address check and JS Recon's endpoint validation use it.

WAF bypass needs proof. To report that the origin bypasses the firewall, a signature probe (an XSS, SQL-injection or path-traversal string) must be blocked at the edge, served with a 2xx by the origin addressed directly, and the origin's page must share at least 80% of its tokens with the page the edge serves for a normal request. A redirect to a login, a 404 or a default page is the origin refusing, not a bypass.

Nuclei's filter. A Nuclei hit whose response mentions a rate limit is dropped; an injection-class hit whose response looks like a firewall block, with a 403 at the start, is dropped too. Only then, if you turned it on, is a model asked (Chapter). Challenge pages from Cloudflare or Akamai are never read as the site's answer: the host is recorded as not really checked.

One writer, in order. With domain discovery in the scan modules (the default), each domain group (or IP-mode run) gets a fresh background writer: a single worker thread. Queuing a write first takes a deep copy of the results on the main thread, so the pipeline can keep changing its dictionary while the writer works on a frozen snapshot. Writes run first in, first out. Each one opens a new Neo4j client, checks the connection and calls an update_graph_from_* method. At the end of the group the pipeline blocks until the queue is drained.

A failed write does not stop the scan. If Neo4j cannot be reached the write is skipped and logged, and any other error is caught and logged. Nothing is retried. The JSON output file is still written after every group, so the results themselves are never lost.

The tenant key. Every recon node is merged on its natural key plus user_id and project_id, backed by a uniqueness constraint. Relationships are merged after matching both ends with the same tenant keys. Three reference labels are global and shared by every project: CVE, MitreData and Capec, because a CVE is a public record, not something you scanned. Every write also stamps updated_at, which the end-of-run prune relies on.

One master switch. AI_IN_PIPELINE is off by default. When the settings load, the five LLM-capable flags are all set to the master's value, so they never decide anything on their own. Each Jev-only hook also needs its own flag. The default model is claude-opus-4-6.

No model key in the scan container. The recon container never calls a model or TypeSafe itself. It posts to the agent's /llm/* or /jev/* endpoints with the scanner key; the agent loads the user's provider and makes the call. The agent rate-limits these calls (a bucket of 60 per user and source address, refilling one per second, and 5,000 per user per day).

Never raise, always fall back. When the call fails, times out, returns something unusable or finds the breaker open, every hook falls back to a fixed, safe answer and the scan carries on. For most hooks that is what the pipeline does without AI; the FFuf planner falls back to .bak .old .config .zip, and page typing to the deterministic labels where one applies. Two circuit breakers guard the agent calls, agent_llm and agent_jev: three transient failures in a row (or two rate-limit answers) open them for 120 seconds, and a refused key or a missing TypeSafe key stops them for the rest of the run (Chapter).

Shadow or act

Each Jev-only hook has a rollout mode fixed in code. In act mode its answer changes the scan. In shadow mode it is asked, its answer is compared with what the pipeline does without it, and the comparison is logged, but nothing changes. Today four of the five act; tool health, which judges whether an empty tool result was a transient failure, runs in shadow. Both modes record every decision against a baseline and print a one-line summary, so a hook can be watched before it is trusted.

Each run is its own short-lived container that receives the tool to run, the roots and your typed targets. A graph builder turns the graph back into the dictionary shape the full-pipeline module expects, so the same scanner code runs in both paths. Writes are MERGEs on the same keys, so rerunning a tool updates nodes and creates no duplicates. A tool that works per domain (subdomain discovery, Shodan, URLScan, Uncover, OSINT enrichment, OpenAPI) runs once per root, with a 2-second pause between roots and one retry 30 seconds after a rate limit.

Up to 12 partial runs can run at once on one project, the same tool included. None can start while the full scan is running or paused, and the full scan cannot start while one is still running. Each reserves 768 MiB of memory (supply-chain recon, 1.75 GiB). A partial run applies the node filters at the end, like a full scan, but never prunes findings: it re-checked one tool, not the whole project.

Typed targets are checked at three points: the modal, the orchestrator and the container each check that a typed host is admitted by one of the run's scope groups and that a typed IP range is at most a /24 (IPv6: a /120). The orchestrator also checks hosts against the hard guardrail and ports against 1 to 65535 (the modal checks ports too). Some types are never typed by hand: a root domain would widen the scope, and endpoints, parameters, technologies and certificates are things a scan discovers, not inputs. Generic typed targets are stamped in the graph as a UserInput node that PRODUCED what they led to.

How settings reach the scan. No setting travels in the container's environment: it gets its ids (project, user, run), service addresses and its own credentials, then fetches the project row with a 30-second timeout, starts from the Python defaults and maps each column onto a setting name, then applies, in order: the RoE rate ceiling, the image and file-path sanitizers, the stealth overrides, the AI master switch and the memory governor, which can lower worker counts and result caps when the host is short of memory. If the fetch fails, the scan fails; it never falls back to defaults.

A preset replaces; it does not merge. Applying a preset sets every preset field: the ones it names get its values, and every other field goes back to its default, except the two model pickers, the Jev switches and your own skill toggles, which keep their values when the preset does not name them. Scope, engagement limits and record, uploads, credentials, secrets and escalation switches are never touched; only when you create a project can a built-in preset switch IP mode on or off. 26 presets ship with RedAmon, filtered by target type in the picker. An LLM can also write a preset from a description; its answer passes a schema that drops any key it does not declare and checks each value's type. Value ranges are checked against the settings registry only when a preset is saved or applied over MCP.

Stealth mode rewrites at scan start. When stealth is on, a registry profile switches off the noisiest tools (Masscan, puredns, VHost/SNI, Kiterunner and more), switches on GAU and ParamSpider (archive lookups, though GAU still checks each archived URL against the target, slowly) and the GraphQL check, makes naabu passive and forces fixed low rates and result caps. It is applied when the scan starts, after the preset values were saved: a forced value replaces the saved one, and a cap leaves a lower saved value alone.

Every outside call is classified. A 2xx is OK and a 404 is "no data"; both count as an answer. A 429 is a rate limit. A 402, or a 401/403 on a call that carried a key, is fatal: the key or the plan was refused. Anything else, including network errors and a non-JSON body, is transient. Some providers get their own rules: FOFA and ZoomEye report errors inside a 200 body, and a Shodan 403 means only that endpoint is not in your plan.

The breaker. Five transient failures in a row open most third-party API breakers. Three are enough for the agent, WHOIS, reverse DNS, the subdomain sources, the web archives, npm, the secret validators and Criminal IP. It stays open for 120 seconds, then lets exactly one probe through. A good probe closes it; a failed probe reopens it for twice as long, up to 30 minutes. A 429 pauses every caller for the Retry-After time (2 s without one, 60 s at most) and resends once; a second 429 in a row opens the provider's breaker. A fatal answer stops that provider for the rest of the run. Endpoints share a parent per provider: a rate limit pauses every endpoint of the provider, a refused key (once no pooled key is left) stops them all for the run, and a transient failure stops only the endpoint that failed.

Key pools. You can give a provider several API keys. The pool rotates to the next key every N calls (10 by default; only received responses count). A fatal answer drops the current key and retries the same request with the next one; a 429 never retires a key. When the last key is refused, the provider is fatal for the run. Hosts get similar treatment from the in-process checkers (security checks, JS recon, jsluice, OpenAPI, GraphQL and others): three connection failures mark a host down, these checkers and the crawlers that follow them skip it, and the prune keeps its findings, because nothing re-checked them.

Coverage first. Each group stamps a coverage record on its domain: which sources a circuit breaker or a crashed phase cut short, and which hosts were skipped as unreachable. If that record cannot be written, the prune does not run at all.

The prune runs only after the run reports success: a single domain whose group finished, a batch in which no group failed, or an IP-mode run. It considers findings from recon sources whose updated_at is older than the run's start, which means this run did not report them again. A source that was cut short is left out, and so is every finding on a host that was skipped as unreachable. Of what is left, a finding a person (or an MCP agent acting for them) muted or judged, or one marked confirmed or proved, is kept and marked stale; everything else is deleted. A stale finding that a later run reports again loses its stale mark when that run's prune runs and the finding's source was not cut short.

Node filters are your mute rules, applied to this run's findings when the rules are active on new scans. They run in a finally block, so they apply even when the run failed. A rule never changes a mute a person or an MCP agent set. It never mutes a finding with a human verdict, a confirmation or a proof, or one a person unmuted, and it releases its own earlier mute on such a finding at the next sweep that reaches it: a scan that reports it again, or Apply to current graph. The rules are re-read at sweep time, so the latest version you saved is the one applied.

New version or overwrite. By default every full scan saves the current graph and opens the next version, which the scan then fills. Overwrite skips the copy and reuses the current version; over MCP it needs a token with the recon:overwrite scope. An empty graph is never copied: the current version is simply reused. A copy larger than 128 MiB compressed is refused, and so is the start, so a very large project can only rescan in overwrite mode.

A refused start costs nothing. If the orchestrator refuses the start, the new version is deleted and the old one becomes current again. If it does not answer at all, the version is kept and the start is reported as unknown (HTTP 503), because the scan may already be running and the copy may be the only one left.

Retention. Only after a start is accepted, old versions are trimmed: the 20 newest unpinned past versions are kept. The current version and pinned ones are never deleted, nor is one a queued or running scan still points to. Failed run records older than 30 days are removed too.

Restoring a version needs a stored copy and nothing else writing the graph: no scan or partial recon, no agent session or triage run, no other scanner, no node-filter apply. The live graph is first saved into the current version, then cleared, keeping the agent's session nodes, and the chosen version is written back.

What readers see during a rescan

Between the clear and the last group, the live graph is half built: hosts are missing because they have not been rebuilt yet, not because they are gone. MCP's graph_summary reports the graph's state (stable, scan_running, agent_writing, activating or unknown) and warns that counts are not settled unless it is stable; the saved version is the settled picture. Deleting an asset also deletes its links, so a kept finding is linked again only when a stage reports it again, and findings other scanners wrote (GVM, for example) stay detached from the rebuilt hosts until those scanners run again.

Who reads the results. The AI agent queries the graph, never the muted findings: a query that names the mute label is refused. GVM reads the recon result file for its targets. The AI attack-surface scan starts from the endpoints recon flagged as AI interfaces. Triage, reports, Recon Delta (which compares two versions) and the project export all read the graph or the saved versions.

Four ways in. The Start button refuses while another scan, a partial recon or a node-filter apply is writing the graph. MCP start_recon also refuses during an agent session, a triage run or another scanner's run, and allows one start per project every 5 minutes. A schedule turns a temporary refusal into a queue entry. The queue retries on its own.

The queue. When a start is refused for lack of memory or a concurrency slot, the UI offers to add the scan to the queue. Each entry has a priority: 10 for one you added, 0 for a schedule, -10 for a batch. Every 20 seconds, and as soon as a finished scan frees memory, a dispatcher walks the ready entries by priority, then age. A capacity wait (memory, a limit, a version being restored, the project busy) puts the entry back for 20 seconds without counting as an attempt. Any other refusal, such as a start outside the Rules of Engagement window, fails the entry. If the scan settings changed since the entry was queued, it waits for you to confirm (needs review); while the agent is working on the project, a full recon waits, because it would clear the graph under it.

Schedules run once, at an interval (15 minutes at least) or on a cron expression. A scheduler checks every 60 seconds for runs that are due, at most 25 at a time. The next run is computed from the moment one fires, so missed runs collapse into one. A temporary refusal becomes a queue entry; a scope that can no longer be scanned pauses the schedule; any other refusal is recorded as a failed run.

The web app checks that nothing else is writing the project's graph (a scan already running or a partial recon), that the scope is valid and that the engagement has no blockers. It then saves the current graph as a version (unless you chose to overwrite it; any refusal further on rolls the copy back) and asks the orchestrator to start, with a 30-second timeout.

The orchestrator does not trust the scope it was sent. It fetches the project again itself, and if the web app does not answer it refuses (fails closed). It checks every name in the scope (hosts, scan roots and their organisations) against the hard guardrail, a fixed list of government, military, education and intergovernmental domains that cannot be switched off, and checks the Rules of Engagement time window. It refuses a second scan or a running partial recon again, as a backstop. Last, the job has to fit in the memory budget.

The container starts from the recon image on the host network, not privileged, with Docker's default capabilities (which include NET_RAW for the raw-socket scanners). Its entrypoint checks the Docker broker socket (up to 8 tries, one second apart), refreshes the DNS resolver list if it is older than 7 days, checks 15 tool images and pulls the 13 registry ones that are missing (the 2 local builds only get a warning), then starts the pipeline. Scanners such as naabu, httpx and nuclei run as sibling containers through that broker; nmap and masscan run inside the recon container itself. Exit code 0 is recorded as completed; anything else is error.

Will it fit in memory?

Every running scan holds a memory reservation: 2 GiB for a full recon by default. Before a new one starts, the orchestrator's admission ledger checks the host. This calculator runs the same rules, with the split redamon.sh sets up at install time: 8% of RAM is kept for the operating system and 65% of the rest for RedAmon's services, which leaves the scan pool.

Then the scan shrinks to fit

Admission reserves memory for the scan; it does not size the scan. That happens inside the container, once, when the settings are loaded: a memory governor compares the host's available memory with its total and scales down 45 thread and pool settings and 20 result caps. Above half the memory free nothing changes. Below that, thread counts shrink in proportion, down to 15% of their value, and result caps are cut to what fits in a tenth of the available memory. It never raises a value, and each cut is logged as [RESOURCE-CAP]. So two scans of the same project can run with different limits, and a crawl that stops at fewer URLs than you set may simply have run on a busy host.

While it runs

The orchestrator streams the container's log to whoever has it open and turns log lines into a phase label with a table of regular expressions, first match wins. The label is a heuristic: the background writer prints its own lines, so the label can jump back to an earlier phase, and it is only updated while someone is reading the log.

Stop, Pause and restarts

Stopping or pausing a scan is not as clean as the buttons suggest, because the work is spread over several containers and the end-of-run steps live inside the pipeline itself.