Knowledge Base & Web Search

The Knowledge Base (KB) adds offline, RAG-enhanced search to the RedAmon agent. Instead of relying solely on Tavily web search, the agent queries a local vector index (FAISS) and graph database (Neo4j) populated with curated security datasets. When the local KB produces high-confidence results, Tavily is skipped entirely. When confidence is low, KB and Tavily results are merged automatically.

For the full technical reference (ingestion pipeline internals, Neo4j schema, security model, on-disk layout, and more), see the Knowledge Base Technical Reference.

For structured CVE queries, prefer cve_intel instead of web_search(include_sources=["nvd"]). The KB's NVD source is good for prose descriptions; cve_intel queries the live ProjectDiscovery vulnx dataset and gives structured fields (EPSS, KEV flag, Nuclei template availability, public PoC links) that semantic search cannot reliably extract.

How It Works (The Big Picture)

The KB has two phases:

1. Ingestion (offline, batch) -- downloads security datasets, chunks them, generates vector embeddings, and stores vectors in FAISS and chunk metadata/content in Neo4j
2. Query (runtime, per-request) -- when the agent calls web_search, it queries the local KB first and only falls back to Tavily if needed

The agent does not interact with the KB directly. It calls web_search() as before, and the tool transparently queries KB first.

What Changed in web_search

Previously, every web_search call went straight to the Tavily API (external web search). Now the tool follows a KB-first strategy:

This means:

• Faster responses: local KB queries complete in ~250ms vs 1-3s for Tavily
• Works offline: if Tavily API is down or no API key is configured, the KB still provides results
• Better pentesting context: curated security data (GTFOBins, LOLBAS, NVD, ExploitDB, OWASP, Nuclei) is more relevant than generic web results for security tasks

Fallback Cascade

The Query Pipeline (6 Stages)

When web_search queries the KB, it runs this pipeline:

1. Hybrid Candidate Retrieval
   - FAISS vector search (cosine similarity, ~30ms)
   - Neo4j fulltext search (Lucene keyword match, ~5ms)
   - Reciprocal Rank Fusion (RRF) merges both ranked lists

2. Filter & Fetch
   - Fetch full metadata from Neo4j
   - Apply source filters (include_sources / exclude_sources)
   - Apply CVSS / severity filters if specified

3. Source Boosts
   - Multiply scores by per-source boost factors
   - tool_docs: 1.20, gtfobins: 1.15, lolbas: 1.15,
     owasp: 1.05, nuclei: 1.00, nvd: 0.90, exploitdb: 0.85

4. Cross-Encoder Rerank
   - BAAI/bge-reranker-base scores top 30 candidates
   - More accurate than vector similarity alone

5. MMR Diversity Filter
   - Avoids returning 5 results from the same source
   - Balances relevance vs variety (lambda=0.65)

6. Sufficiency Check
   - If top score >= 0.35: return KB results
   - Otherwise: fall back to Tavily

Data Sources

The KB ingests from seven curated security datasets, organized in three profiles:

With GPU or API embeddings, all profiles complete in minutes.

Source-Specific Parameters

The agent can target specific sources using web_search parameters:

When Does Ingestion Run?

There are four ways ingestion can start:

1. Automatic on Install / Up / Restart

The KB is opt-in. Pass --kbase once at install time to enable it; the choice is persisted (via the .kbase-enabled flag file) and respected by subsequent update / up / restart runs without re-passing the flag.

./redamon.sh install --kbase   # opt in: builds lite profile after install
./redamon.sh up                # rebuilds lite profile if KB was enabled at install
./redamon.sh restart           # rebuilds lite profile if KB was enabled at install

./redamon.sh install (no flag) skips the KB entirely and runs in Tavily-only mode.

When KB is enabled, RedAmon automatically:

1. Checks if KB is enabled (KB_ENABLED env var, set to false when --kbase was not passed at install)
2. Runs make kb-build-lite inside the agent container
3. Downloads and indexes the lite sources (~30-60 seconds)
4. If ingestion fails, the agent starts anyway in Tavily-only mode (non-fatal)

2. Manual CLI Commands

# First-time build with a specific profile
make kb-build-lite              # fast, no API keys needed
make kb-build-standard          # adds NVD CVEs (~6-8 min)
make kb-build-full              # adds Nuclei templates (~10-15 min)

# Incremental updates (only fetches changed content)
make kb-update-nvd              # recommended: daily
make kb-update-exploitdb        # recommended: weekly
make kb-update-nuclei           # recommended: weekly
make kb-update-gtfobins         # recommended: monthly
make kb-update-lolbas           # recommended: monthly

# Or via redamon.sh
./redamon.sh kb build standard
./redamon.sh kb update nvd
./redamon.sh kb rebuild full    # wipe and rebuild from scratch

3. Automated Refresh Sidecar (Opt-in)

A background container that keeps the KB fresh on a schedule:

Enable it with:

KB_REFRESH_ENABLED=true docker compose --profile kb-refresh up -d kb-refresh

4. Direct Python (Local Development)

python -m knowledge_base.curation.data_ingestion --profile lite --neo4j-uri bolt://localhost:7687

Embedding: CPU vs GPU vs API

The KB uses a sentence-transformer model (intfloat/e5-large-v2, ~1.3 GB) to generate vector embeddings. How this model runs depends on your hardware.

How Device Selection Works

The underlying sentence-transformers library automatically detects available hardware:

Key points:

• You do not need to configure anything for CPU/GPU. The library detects CUDA automatically. If you have an NVIDIA GPU with CUDA drivers installed, it will be used. Otherwise, CPU is used.
• Query latency is always fast (~30ms) regardless of CPU or GPU, because the model stays loaded in memory after the first query. The difference only matters during ingestion when thousands of chunks need embedding.
• The model is pre-cached in the Docker image (~1.3 GB for the embedder + ~568 MB for the reranker). No download happens at runtime -- it was downloaded during docker compose build agent.

When to Use API Mode

If your machine has no GPU and limited CPU, you can offload embedding to an external API. This speeds up ingestion significantly but adds network latency to every query.

Configure in .env:

# Switch to API-based embedding
KB_EMBEDDING_USE_API=true
KB_EMBEDDING_API_KEY=sk-your-openai-key
KB_EMBEDDING_API_MODEL=text-embedding-3-small
# KB_EMBEDDING_API_BASE_URL=  (leave empty for OpenAI, or set for compatible APIs)

Using OpenAI-Compatible APIs

The openai provider works with any API that implements the OpenAI embeddings endpoint. Set KB_EMBEDDING_API_BASE_URL to point to the compatible server:

Example with Ollama (free, local):

KB_EMBEDDING_USE_API=true
KB_EMBEDDING_API_MODEL=nomic-embed-text
KB_EMBEDDING_API_KEY=ollama
KB_EMBEDDING_API_BASE_URL=http://host.docker.internal:11434/v1

Important: Ingestion and query must use the same embedding model. If you switch between local and API mode (or change the API model), you must rebuild the entire KB index:

make -C services/knowledge_base kb-rebuild-lite MODE=docker    # or standard/full

CPU-Only Tips

If you are running on CPU only:

1. Start with the lite profile -- it indexes ~2,000 chunks in 30-60 seconds even on CPU
2. Use incremental updates (kb-update-*) instead of full rebuilds -- only changed content gets re-embedded
3. Consider API mode if you plan to use the standard or full profiles regularly
4. Query performance is not affected -- once the model is loaded, single-query embedding is fast on any hardware

Typical Workflows

Workflow 1: Fresh Install (Default)

./redamon.sh install

What happens behind the scenes:

1. Docker containers start (agent, neo4j, webapp, ...)
2. KB lite build runs automatically inside the agent container
3. Downloads GTFOBins, LOLBAS, OWASP, ExploitDB tarballs/CSV
4. Chunks, embeds, and indexes ~2,000 chunks
5. Agent starts with KB loaded -- web_search now queries KB first

Result: web_search uses KB for security queries, Tavily for everything else.

Workflow 2: Upgrading to Standard Profile

# Add NVD CVE data (optional: set API key for 10x faster ingestion)
echo "NVD_API_KEY=your-key" >> .env
make kb-build-standard
docker compose restart agent

Now web_search also searches recent high-severity CVEs (CVSS >= 7, last 90 days).

Workflow 3: Keeping Data Fresh

# Quick daily update (just NVD)
make kb-update-nvd

# Or enable the automatic refresh sidecar
KB_REFRESH_ENABLED=true docker compose --profile kb-refresh up -d kb-refresh

No agent restart needed after incremental updates -- the FAISS index is reloaded on the next query.

Workflow 4: Switching to API Embeddings

# 1. Copy the template and configure
cp .env.example .env
# Edit .env and set:
#   KB_EMBEDDING_USE_API=true
#   KB_EMBEDDING_API_KEY=sk-your-key
#   KB_EMBEDDING_API_BASE_URL=  (leave empty for OpenAI, or set for compatible APIs)

# 2. Rebuild the index (required when changing embedding model)
make -C services/knowledge_base kb-rebuild-lite MODE=docker

# 3. Restart agent to pick up new env vars
docker compose restart agent

Workflow 5: Disabling KB for a Specific Project

In the project settings UI, set KB Enabled to false. The agent will use Tavily-only for that project without affecting other projects.

Workflow 6: Targeting Specific Sources

When chatting with the agent, it automatically decides which sources are relevant. But you can also configure per-project source filters:

Troubleshooting

KB not loading?

# Check if KB is enabled
docker exec redamon-agent env | grep KB_ENABLED

# Check if index files exist
ls -la services/knowledge_base/data/index.faiss services/knowledge_base/data/chunk_ids.json

# Check agent logs
docker logs redamon-agent 2>&1 | grep -i "knowledge\|kb\|faiss"

Rebuild from scratch

make kb-rebuild-lite    # or standard/full
docker compose restart agent

Check index stats

make kb-stats

Test a KB query manually

docker exec -it redamon-agent python -c "
import os
from knowledge_base import PentestKnowledgeBase
from knowledge_base.faiss_indexer import FAISSIndexer
from knowledge_base.neo4j_loader import Neo4jLoader
from knowledge_base.embedder import Embedder
from neo4j import GraphDatabase

embedder = Embedder('intfloat/e5-large-v2')
faiss = FAISSIndexer('/app/knowledge_base/data', dimensions=1024)
driver = GraphDatabase.driver('bolt://neo4j:7687', auth=('neo4j', os.environ['NEO4J_PASSWORD']))  # generated on fresh install; see .env
kb = PentestKnowledgeBase(faiss, Neo4jLoader(driver), embedder)
kb.load()

results = kb.query('sudo privilege escalation linux', top_k=5)
for r in results:
    print(f'{r[\"score\"]:.3f} [{r[\"source\"]}] {r[\"title\"]}')
"

Full technical documentation: Knowledge Base Technical Reference -- covers ingestion pipeline internals, Neo4j graph schema, security model, on-disk layout, incremental caching, and all configuration options.