OwnSearch

OwnSearch

Self-hosted MCP and HTTP search over a scored fleet of public SearXNG instances.

mcp-name: io.github.devops-infinity/ownsearch

Status: pre-release, workspace version 0.1.0.

OwnSearch discovers public SearXNG instances from searx.space, probes them at a low rate, scores them, keeps the fleet in a local SQLite database, rotates the starting instance, retries a failing one, and answers from the next healthy instance. Each result carries the sources that supplied it, with the instance and, when known, the engine. It runs as a Model Context Protocol (MCP) server over stdio or Streamable HTTP, and the HTTP port also answers two REST calls.

Owner: Md. Sazzad Hossain Sharkar.

Reference material: docs/scanner-policy.md for instance policy, docs/operations.md for the runbook, RELEASING.md for the release procedure, SECURITY.md for the reporting policy.

Contents

• What it does
• What is enabled by default
• What it does not do
• Requirements
• Build and install
• Quick start
• Connect an MCP client
• Tools
• Search behavior
• Scanner policy
• Quality model
• Configuration
• HTTP mode
• Container
• Local files and keep windows
• Database
• Database commands
• AI-assisted database changes
• Development
• Troubleshooting
• Privacy
• Security
• Support
• Contributing
• Distribution
• License

What it does

• Discovers public HTTPS SearXNG instances from https://searx.space/data/instances.json, and imports an embedded snapshot of 67 instances on the first run, before any network call.
• Probes each candidate once per scan: it resolves the host and validates the TLS chain and certificate dates, reads /config, and makes one JSON search. It makes an HTML search only when JSON is denied, and an exact-Google search only when the instance advertises that engine and JSON worked.
• Applies hard gates before scoring, so a certificate near expiry, an analytics tracker, or an unreadable /config keeps an instance out of the active fleet.
• Selects an active fleet with a size limit, a per-network limit, and hysteresis, so one bad scan does not remove a working instance.
• Rotates the starting instance on every search, makes up to three attempts per instance on a network error or 5xx, cools an instance down when it keeps failing, and fails over to the next instance until the search succeeds or the deadline passes.
• Merges results from several instances, keeps every source observation, and marks partial coverage.
• Never retries a 429, never solves or submits a CAPTCHA, and never tries to look like a human browser.

What is enabled by default

These capabilities are on by default. Each one is controlled by the setting named in Configuration.

• HTML search fallback: on. When an instance refuses JSON with a 403, another 4xx or 5xx, or a non-JSON body, OwnSearch reads the ordinary HTML result page instead. A 401, a 429, and a challenge page are not retried through HTML; those follow the quarantine and cooldown rules.
• Round-robin selection: on. Each search starts with a different healthy instance.
• Failover: on. A failing instance gets its configured attempts, then it cools down and the search moves to the next instance.
• Every search engine the selected instance enables: on. OwnSearch sends no engine filter by default, so Google, Bing, DuckDuckGo, Brave, and every other engine the instance runs are available. Set engines on a call to narrow it.
• All SearXNG categories: on. OwnSearch sends no category filter by default.
• Exact-Google probing: on, whenever the instance advertises the google engine.
• Page extraction: on. ownsearch_fetch fetches a public page and returns it as Markdown, plain text, or cleaned HTML, with the title and a size cap.
• JavaScript rendering: on. When a page is an empty shell or the call sets render: "always", a headless Chrome renders it, the browser stays open while it is used, and it closes on idle, on two missed health answers, or at its maximum age.
• Same-host crawling: on. ownsearch_fetch follows up to max_pages - 1 links from the page you name, at most seven, and returns each one under pages. same_host_only defaults to true, so a call never wanders to another site unless you allow it.
• Result attribution, field presets, the response cache, session statistics, and typed database changes (with a preview and a one-use token) are all available without extra flags.

What it does not do

• It does not host or operate a SearXNG instance.
• It does not call a paid search API.
• It does not make your queries anonymous. The operator of each public instance sees them.
• It does not run JavaScript on the search path. The only renderer is the fetch tool's optional headless browser, and it runs only on the page you name.
• It does not bypass a bot check. Many public instances answer automated clients with 429, a challenge page, or a CAPTCHA. OwnSearch records that and moves on.

Requirements

• Rust 1.94.1 or newer to build. The pinned toolchain is in rust-toolchain.toml (1.99.1).
• A Linux or macOS machine. Windows builds are part of the release target list.
• Outbound HTTPS for discovery, probes, and searches.
• A C compiler for the bundled SQLite build: build-essential on Debian and Ubuntu, or the Command Line Tools on macOS.
• A Chrome or Chromium executable for JavaScript rendering. Set OWNSEARCH_BROWSER_PATH when auto-detection misses it.

Build and install

git clone https://github.com/devops-infinity/ownsearch
cd ownsearch
cargo build --release --locked

The binary lands at target/release/ownsearch. To install it into ~/.local/bin after a full gate:

tools/reinstall.sh

Quick start

ownsearch doctor
ownsearch scan                # refresh discovery and probe the fleet
ownsearch scan                # once OWNSEARCH_SCAN_MIN_INTERVAL_SECS (30 by default) has passed, a second pass promotes repeat successes to active
ownsearch serve               # MCP over stdio

A dry run changes no fleet data and prints what would happen. On a first run it may create the empty state schema:

ownsearch scan --dry-run --limit 12

Searches use the active fleet. When no instance is active, OwnSearch falls back to scanned candidates that passed every hard gate and says so in warnings.

Connect an MCP client

{
  "mcpServers": {
    "ownsearch": {
      "command": "/home/you/.local/bin/ownsearch",
      "args": ["serve"],
      "env": {
        "OWNSEARCH_STATE_DIR": "/home/you/.local/state/ownsearch"
      }
    }
  }
}

No API key is needed. The only secret OwnSearch can hold is an HTTP bearer token, used only when you serve Streamable HTTP.

Over Streamable HTTP, point the client at http://127.0.0.1:8774/mcp, or at the same host with no path. Both serve the MCP endpoint, and any other path answers 404 with a problem body. The port also answers /healthz, /startupz, /readyz, and /metrics. Send Authorization: Bearer <token> once OWNSEARCH_BEARER_TOKENS is set. The REST calls POST /v1/search and POST /v1/fetch take the tool arguments as the JSON body.

Tools

• ownsearch_search: federated web search. Arguments: query (required), categories, engines, language, page, time_range (day, week, month, year), safesearch (off, moderate, strict), limit, fields (minimal, standard, full), mode (fast, balanced, exhaustive), timeout_secs, and result_text (summary, full). It may refresh local fleet state before answering.
• ownsearch_fetch: fetch a public page as markdown (default), text, or html, with url required and max_chars from 1000 to 200000. max_pages runs from 1 to 8 and defaults to 3; beyond 1 it follows links from the first page and returns them under pages, staying on the same host unless same_host_only is false. An HTTP redirect is followed up to three hops, each hop passes the same public-address check as the first request, and the answer reports finalUrl plus the redirects chain so a caller never has to guess where a link ended up. An application/pdf answer is parsed under an 8 MiB decompressed-text cap and shaped into Markdown paragraphs. render chooses the browser: auto (the default) renders only a page that looks like an empty JavaScript shell, always renders every call, and never stays on HTTP. Markdown is extracted from the main content: navigation, sidebars, cookie banners, share blocks, comment sections, and footers are pruned, the densest content root wins over link lists, and code fences carry their language. Private and loopback addresses are refused, and every page is capped and sanitized.
• ownsearch_instance_info: lifecycle and capability facts for one instance or a list, including its slug, status, score, rank, version, JSON and HTML access, engines, categories, and exact-Google evidence. Every result source and every scan status change carries the slug; a failed source names its instance id and url. The /config instance name comes from ownsearch_db_query with entity: "capabilities".
• ownsearch_instance_health: hard gate result, certificate days left, TLS protocol and grade, median search time, cooldown, consecutive failures, and the most recent lifecycle events.
• ownsearch_scan: refresh discovery, probe instances, rescore, and reselect the fleet. It defaults to dry_run: true. Within one process, a writing scan runs one at a time per state database, waits OWNSEARCH_SCAN_MIN_INTERVAL_SECS between runs, and refuses a named instance that is not yet due unless the call sets force.
• ownsearch_session: local call counts, cache state, ledger length and head, and fleet size. No network call.
• ownsearch_db_query: bounded read-only queries over instances, quality, policy overrides, capability snapshots, events, snapshots, and discovery runs. Each entity applies its own filters, and a filter that the selected entity cannot apply is refused instead of ignored. Events page in SQL, so any offset is reachable. It never accepts SQL.
• ownsearch_db_crud: one typed lifecycle or policy change. It needs write mode, a reason, and a one-use confirmation token from a dry-run preview. It cannot delete an instance row.

Every tool returns structuredContent plus a one-line text summary. Every input schema sets additionalProperties: false. Every field a preset may drop is optional in the published output schema.

Search behavior

Modes control how many instances answer one query.

• fast targets one instance.
• balanced, the default, targets three.
• exhaustive targets OWNSEARCH_EXHAUSTIVE_MAX.

Failover works the same way in every mode. The selected instances answer in parallel. A network error, a timeout, or a 5xx is retried up to OWNSEARCH_SEARCH_ATTEMPTS times per instance. A 4xx other than 408 and 429, a 403, or a challenge page is never retried. When an instance still fails, OwnSearch sets its cooldown_until, records a search_failed event, and tries the next instance in score order until it reaches its target or the search deadline (OWNSEARCH_SEARCH_DEADLINE_SECS). A failed search changes neither the status nor the probe counters. A scan probe or an operator command moves an instance between active, probation, and quarantined, and only a probe resets the success and failure counts. Failover keeps walking the scored fleet in order until it reaches its target or the deadline, so an answer low in the list is still found. A bounded queue sits in front: OWNSEARCH_SEARCH_MAX_CONCURRENT searches run at once, up to OWNSEARCH_SEARCH_QUEUE_LIMIT wait for a slot for OWNSEARCH_SEARCH_QUEUE_WAIT_SECS, and the rest answer ownsearch.rate_limited without stacking work, which is what keeps memory and CPU bounded on a small machine. An answer with zero results is a success, not a failure; only a page whose results carry no usable title and URL counts as a parser failure.

Round-robin rotates the starting point inside the top healthy window, so two searches in a row do not always begin with the same instance. Turn it off with OWNSEARCH_ROUND_ROBIN=false when you prefer a strict score order.

Before any scan has scored the fleet, or when every scored instance is cooling, a search falls back to the highest ranked discovered instances that have not failed a hard gate, rather than failing with ownsearch.no_fleet, and the response says so in warnings. The response never invents a result: when every attempt is refused or rate limited, sources.failed names each instance, its code, and what it answered. sources.fallbacks names each instance whose answer came from the HTML parser instead of JSON.

Field presets control payload size. minimal keeps title, URL, rank, score, and source id. standard adds the snippet, engines, published date, and the full source list. full keeps everything.

Scanner policy

• A built-in scheduler refreshes discovery at startup and on each discovery window (OWNSEARCH_DISCOVERY_REFRESH_SECS), with a document capped at 500 records. It then probes due instances at the scan interval, one scan at a time, after a jitter delay.
• A pinned instance stays in the selection even when its score would drop it. A pin overrides the score order, the hysteresis, the per-network cap, and a Google requirement, and it lets a candidate or missing record enter the fleet. A failed hard gate still excludes it.
• A discovery record whose host resolves to a loopback, private, link-local, documentation, or reserved address is skipped, and an explicit scan of a denied, disabled, retired, or cooling instance is refused.
• One probe per kind per instance per run.
• Probes run four at a time by default, with jitter. OWNSEARCH_SCAN_CONCURRENCY changes the count for any run, scheduled or on demand.
• A 429 starts a cooldown and keeps the instance's status. A 401 or a challenge page quarantines it at once. A 403 counts as a failed scan: it reschedules the scan and moves the instance to probation or quarantine once the failure count passes its threshold. Repeated failures also lower its score.
• The user agent names the product: OwnSearch (MCP fleet scanner; +https://github.com/devops-infinity/ownsearch) v<version>.
• No proxy rotation, no browser automation against instances, no header spoofing.

The full policy is in docs/scanner-policy.md.

Quality model

Hard gates, in order. A failure keeps the instance out of the active fleet.

• HTTPS only.
• A valid certificate chain with a matching hostname.
• At least OWNSEARCH_CERT_MIN_DAYS days left on the certificate.
• No analytics tracker.
• A parseable /config.
• Exact Google present, but only when OWNSEARCH_GOOGLE_REQUIRED=true.

Score out of 100:

• Monthly uptime, 20 points.
• Yearly uptime, 10 points.
• Search success rate, 20 points.
• Median search time, 15 points.
• Initial response success and time, 10 points.
• Exact Google success and timing, 10 points.
• TLS grade, protocol, and certificate lifetime, 10 points.
• SearXNG version recency, 5 points.
• Bounded modifiers, up to 5 points, for engine coverage, category coverage, repeated successes, repeated failures, and parser drift.

Fleet selection sorts by score, takes at most OWNSEARCH_FLEET_MAX instances, allows at most OWNSEARCH_MAX_PER_NETWORK instances from one announced network prefix, excludes Tor records, and excludes an instance without exact Google support when OWNSEARCH_GOOGLE_REQUIRED is set, unless the instance is pinned. Hysteresis needs OWNSEARCH_PROMOTE_SUCCESSES good scans to promote, OWNSEARCH_PROBATION_FAILURES failed scans to demote, and OWNSEARCH_QUARANTINE_FAILURES failed scans to quarantine. A recovered missing instance returns through probation.

Configuration

Settings resolve in this order: CLI flags, OWNSEARCH_* environment variables, a TOML profile at ${XDG_CONFIG_HOME}/ownsearch/profile.toml, then built-in defaults.

Most environment variables have a matching profile key: drop the OWNSEARCH_ prefix and lowercase the rest. For example OWNSEARCH_FLEET_MAX is fleet_max, and OWNSEARCH_STATE_DIR is state_dir. Three keys do not follow the pattern: OWNSEARCH_BROWSER_PATH is browser_executable, and OWNSEARCH_BEARER_TOKENS and OWNSEARCH_PROFILE have no profile key. Two keys exist only in the profile: cache_max_bytes and preset.

Paths and transports

• OWNSEARCH_PROFILE: path to the TOML profile.
• OWNSEARCH_STATE_DIR: directory for instances.db, the snapshot, events, and backups. Default ${XDG_STATE_HOME}/ownsearch.
• OWNSEARCH_LEDGER_DIR: directory for calls.jsonl, its archives, and args.key. Default ${XDG_DATA_HOME}/ownsearch/ledger.
• OWNSEARCH_HTTP: serve Streamable HTTP instead of stdio. Default false.
• OWNSEARCH_BIND: HTTP bind address. Default 127.0.0.1:8774.
• OWNSEARCH_BEARER_TOKENS: comma-separated bearer tokens, each at least 32 characters.
• OWNSEARCH_ALLOWED_HOSTS: comma-separated Host values for HTTP mode.
• OWNSEARCH_ALLOWED_ORIGINS: comma-separated Origin values for HTTP mode.
• OWNSEARCH_LOG_FORMAT: text or json. Logs go to stderr only.
• OWNSEARCH_ALLOW_PRIVATE_INSTANCES: accept loopback and private addresses for discovery records, probes, search, and fetch. Default false, and it exists for local testing only. Leave it off in any deployment that can reach the internet, because with it on a hostile discovery document can point the scanner at a local service.

Discovery and scanning

• OWNSEARCH_DISCOVERY_URL: discovery source. Default https://searx.space/data/instances.json.
• OWNSEARCH_DISCOVERY_REFRESH_SECS: refresh window. Default 21600.
• OWNSEARCH_OFFLINE: skip discovery and use local state only. Default false.
• OWNSEARCH_SCAN_CONCURRENCY: parallel probes, 1 to 8. Default 4.
• OWNSEARCH_SCAN_JITTER_MS: random delay between probes, up to this many milliseconds. Default 250.
• OWNSEARCH_PROBE_TIMEOUT_SECS: per-request timeout for each probe step. Default 15; one probe stops after five times this value.
• OWNSEARCH_SCAN_DEADLINE_SECS: whole-scan deadline. Default 600, from 30 to 7200. The scan stops with a warning when it passes.
• OWNSEARCH_SCAN_MIN_INTERVAL_SECS: minimum gap between two writing scans on the same state database. Default 30, from 0 to 3600, and 0 turns the gap off. One scan at a time within a process always applies.
• OWNSEARCH_FETCH_DEADLINE_SECS: whole-fetch deadline, including followed links and a browser render. Default 60, from 5 to 600.
• OWNSEARCH_BROWSER: render JavaScript pages in headless Chrome. Default true.
• OWNSEARCH_BROWSER_PATH: Chrome or Chromium executable. Empty uses auto-detection.
• OWNSEARCH_BROWSER_IDLE_SECS: how long an idle browser stays open. Default 120, from 30 to 3600.
• OWNSEARCH_BROWSER_NO_SANDBOX: pass --no-sandbox, required inside a container that cannot set the sandbox. Default false.
• OWNSEARCH_BROWSER_WATCHDOG_SECS: how often the browser this process launched is checked for an answer. Default 30, from 5 to 600. Two missed answers close it.
• OWNSEARCH_BROWSER_MAX_LIFETIME_SECS: how long one browser session may live before it is replaced. Default 1800, from 60 to 86400.
• OWNSEARCH_REQUEST_TIMEOUT_SECS: HTTP client timeout. Default 30.
• OWNSEARCH_CONNECT_TIMEOUT_SECS: connect timeout. Default 5.
• OWNSEARCH_MAX_UPSTREAM_BYTES: response body cap. Default 4194304.
• OWNSEARCH_COOLDOWN_SECS: cooldown after a rate limit, a challenge, or a search failure. Default 3600.
• OWNSEARCH_PROMOTE_SUCCESSES, OWNSEARCH_PROBATION_FAILURES, OWNSEARCH_QUARANTINE_FAILURES: hysteresis. Defaults 2, 2, 4.
• OWNSEARCH_FLEET_MIN, OWNSEARCH_FLEET_MAX: fleet size, defaults 6 and 12.
• OWNSEARCH_MAX_PER_NETWORK: instances per announced network prefix, default 2.
• OWNSEARCH_CERT_MIN_DAYS: certificate lifetime gate, default 14.
• OWNSEARCH_HTML_FALLBACK: HTML probe and fallback. Default true.
• OWNSEARCH_GOOGLE_REQUIRED: exact Google as a hard gate. Default false.

Search

• OWNSEARCH_SEARCH_DEADLINE_SECS: whole-search deadline, default 30.
• OWNSEARCH_EXHAUSTIVE_MAX: instances in exhaustive mode, default 16.
• OWNSEARCH_SEARCH_ATTEMPTS: attempts per instance, 1 to 5. Default 3.
• OWNSEARCH_SEARCH_FAILOVER: fail over to the next instance. Default true.
• OWNSEARCH_ROUND_ROBIN: rotate the starting instance. Default true.
• OWNSEARCH_RESULT_CAP: default result limit and the ceiling on a call's limit. Default 25, from 1 to 100.
• OWNSEARCH_SEARCH_MAX_CONCURRENT: searches that run at once, default 8.
• OWNSEARCH_SEARCH_QUEUE_LIMIT: searches allowed to wait for a slot, default 64.
• OWNSEARCH_SEARCH_QUEUE_WAIT_SECS: how long a queued search waits before it answers ownsearch.rate_limited, default 30.
• OWNSEARCH_CACHE_TTL_SECS: how long a cached answer stays fresh. Default 300, and 0 turns the cache off.
• Profile key cache_max_bytes: cache byte cap. Default 16 MiB.
• Profile key preset: default field preset, minimal, standard, or full. Default standard.

Keep windows and maintenance

• OWNSEARCH_KEEP_WINDOW: how long to keep ledger archives, event lines, and pre-migration backups. Accepts 90s, 15m, 2h, 3d, or plain seconds. Default 3d. Minimum 1 hour, maximum 90 days, and a negative or malformed value is refused at startup.
• OWNSEARCH_MAINTENANCE_INTERVAL_SECS: how often the running server runs the cleaner. Default 3600, minimum 60, maximum 86400.
• OWNSEARCH_WATCHDOG_SECS: how long the process may go without progress before it exits with code 75 so a supervisor restarts it. Default 120, from 0 to 3600, and 0 turns it off.

The cleaner runs at startup, after every applied scan, on that interval while a server runs, and inside ownsearch db maintenance. It rotates the live ledger when its oldest row passes the keep age, deletes ledger archives and pre-migration-*.db files older than the window, and trims old lines from instances.events.jsonl. The trim holds a lock file, so an event written at the same moment is never lost. The same interval also expires old rows in SQLite: probes, TLS observations, capability snapshots, metrics, snapshots, events, and discovery runs each keep their own window, and ownsearch db maintenance reports what it removed. Two processes on one state database take turns through a lock file, so a sweep never runs twice. If a file cannot be removed, the run logs a warning that names ownsearch db maintenance as the manual repair, and a sweep that reports a problem writes an attention row to maintenance_runs.

AI database changes

• OWNSEARCH_AI_DB_MODE: read-only or read-write. Default read-write, because every change still needs a preview and a one-use token; set read-only to refuse all typed database changes.
• OWNSEARCH_AI_DB_MAX_CHANGES_PER_CALL: field changes per call, default 5.
• OWNSEARCH_AI_DB_CONFIRM_TTL_SECS: confirmation lifetime, default 120.
• OWNSEARCH_AI_DB_MIN_INTERVAL_SECS: minimum gap between applies, default 2.

Profile example

version = 1
fleet_min = 6
fleet_max = 12
html_fallback = true
google_required = false
result_cap = 10
search_attempts = 3
round_robin = true
keep_window = "3d"
ai_db_mode = "read-only"

HTTP mode

OWNSEARCH_BEARER_TOKENS="$(openssl rand -hex 24)" ownsearch serve --http

Any random value of at least 32 characters works; openssl rand -hex 24 is one way to make one.

• The default bind is 127.0.0.1:8774.
• A non-loopback bind without a bearer token is refused at startup.
• Tokens need at least 32 characters and are compared in constant time.
• Host is checked against a loopback list or OWNSEARCH_ALLOWED_HOSTS.
• An Origin header is rejected unless it matches OWNSEARCH_ALLOWED_ORIGINS.
• Request bodies are capped at 1 MiB, headers must arrive within 10 seconds, and concurrent requests are capped.
• GET /healthz is liveness. GET /startupz runs a quick database check on every request and answers 503 while the check fails. GET /readyz runs a cheap liveness read on every request and fails during shutdown, so a monitor never waits on a whole-database scan. GET /metrics reports call counters, per-tool counts, and per-route HTTP counts with a latency sum and maximum.
• Middleware runs outermost first: body deadline, request id, origin check, host check, bearer check, concurrency gate, body limit, trace layer, metrics, route. Security headers wrap the whole router. The trace layer logs method, path, status, and latency.
• Shutdown flips readiness first, then drains in-flight calls under a deadline.
• Under a supervisor, a hung process is replaced. OWNSEARCH_WATCHDOG_SECS bounds how long it may stall before it exits with code 75, and docker-compose.yml sets restart: unless-stopped with a 30 second stop grace period.

REST endpoints

The same engine answers two JSON endpoints on the HTTP port, behind the same bearer check, origin check, host check, and body limit as the MCP stream.

• POST /v1/search takes the fields of ownsearch_search, with query required.
• POST /v1/fetch takes the fields of ownsearch_fetch, with url required.

curl -s http://127.0.0.1:8774/v1/search \
  -H 'content-type: application/json' \
  -d '{"query":"rust programming language","limit":5}'

A successful call answers 200 with the structured content the MCP tool returns. A refused call answers 400 for bad arguments, 401 for a missing or wrong token, 403 for a policy refusal, 404 for a missing route or resource, 405 for a wrong method, 409 for an expired confirmation, 429 for rate limiting, 502 for an upstream failure, 503 for no fleet or a shutdown, and 504 for a timeout, with an RFC 9457 application/problem+json body that carries type, title, status, detail, instance, and the stable code and remedy as extension members, and no internal detail. Every REST call is recorded in the ledger and counted in /metrics exactly like an MCP call.

Put Caddy in front for any remote use. Never expose an anonymous scanner.

Store the token list in the OS keychain instead of the environment:

printf '%s' "$TOKEN" | ownsearch config set-bearer-token

Container

docker build -t ownsearch .
docker run --rm -p 127.0.0.1:8774:8774 \
  -e OWNSEARCH_BEARER_TOKENS="$(openssl rand -hex 24)" \
  -v ownsearch-state:/home/ownsearch/.local/state/ownsearch \
  ownsearch

docker-compose.yml holds the same setup with restart: unless-stopped, both state volumes, and a healthcheck. The image sets OWNSEARCH_BIND=0.0.0.0:8774, a ledger directory under /home/ownsearch/.local/share/ownsearch/ledger, and OWNSEARCH_BROWSER=false, so a container renders no page. OWNSEARCH_BEARER_TOKENS is required at run time and the server refuses to start without it. Add OWNSEARCH_ALLOWED_HOSTS with the names clients use, or the Host check refuses every request that does not name localhost or the bind address. The state volume keeps the database and snapshot across restarts; mount the ledger directory too when the ledger and args.key must survive a restart.

Local files and keep windows

• Database: ${XDG_STATE_HOME}/ownsearch/instances.db, or OWNSEARCH_STATE_DIR/instances.db.
• Snapshot: instances.snapshot.json beside it, regenerated after each discovery run, each scan, each AI change, and the bootstrap import.
• Discovery history: instances.events.jsonl, trimmed to the keep window.
• Backups you create: backups/instances-<stamp>.db, written with VACUUM INTO, verified with a quick check and a foreign key check, and removed after the keep window like every other temporary file.
• Pre-migration backups: pre-migration-<stamp>.db, removed after the keep window.
• Browser profiles: browser/profile-<pid> under the state directory, owner-only, removed when the session closes, and reaped at startup when the process that owned one is gone.
• Ledger: calls.jsonl, archived to calls-<stamp>.jsonl at the keep boundary, and archives older than the window are removed.
• Arguments key: args.key beside the ledger. It keys the argument hashes and the ledger chain, so it is kept.

The state directory is created with mode 0700 and files with mode 0600. WAL mode adds instances.db-wal and instances.db-shm. Database rows have their own windows, applied by ownsearch db maintenance: probes 90 days, TLS observations 90 days, capability snapshots 90 days, hourly metrics 180 days, fleet snapshots 30 days, discovery runs 180 days, lifecycle events 365 days. Instance rows are never deleted by the scanner.

Database

The database is SQLite in WAL mode with foreign keys on, STRICT tables, and a checksummed migration ledger. One migration file holds the whole schema: tables, foreign keys, checks, indexes, and views, so a development build always creates the current shape in one step. The column-level schema is crates/ownsearch-core/src/db/migrations/0001_initial.sql; this page does not repeat it.

Tables: schema_migrations, application_metadata, discovery_sources, discovery_runs, instances, capability_snapshots, engines, categories, capability_engines, capability_categories, probes, tls_observations, hourly_probe_metrics, instance_quality, instance_events, fleet_snapshots, instance_policy_overrides, and maintenance_runs. Two views serve the hot paths: active_fleet and scan_due.

ownsearch db verify checks the file's integrity and foreign keys and prints the schema version. The store also checks each migration checksum when it opens. A backup is written with VACUUM INTO and verified before it is used.

Database commands

• ownsearch db path
• ownsearch db verify
• ownsearch db backup [--out PATH]
• ownsearch db export [--out PATH]
• ownsearch db maintenance (WAL checkpoint, vacuum, data expiry, and the cleanup pass)
• ownsearch db counts [--json]
• ownsearch instances list [--status S] [--limit N] [--json]
• ownsearch instances show [ID] [--url URL] [--json]
• ownsearch instances add URL [--pin] [--json]
• ownsearch instances pin ID, unpin ID, deny ID [--reason TEXT], reset-status ID
• ownsearch instances retire ID [--reason TEXT], restore ID

Your own instance

ownsearch instances add http://localhost:8888/ --pin stores a SearXNG instance you run yourself, whether it listens on a localhost port, a Docker network address, or any other private address. The instance starts as a candidate. A scan probes it like any other instance, and two passing probes promote it to active, after which searches use it. A loopback or private host skips the TLS and certificate gate, because those checks describe a public host. The address rule still applies: set OWNSEARCH_ALLOW_PRIVATE_INSTANCES=true, and keep the instance on a network you control, since the same setting lets a hostile discovery document point the scanner at a local service.

Instance CRUD examples

Create an instance:

ownsearch instances add http://127.0.0.1:8080/ --pin

Read it back:

ownsearch instances list --limit 10
ownsearch instances show 1
ownsearch instances show --url http://127.0.0.1:8080/

Update its policy:

ownsearch instances pin 1
ownsearch instances unpin 1
ownsearch instances deny 1 --reason "host is gone"
ownsearch instances reset-status 1

Delete it from the fleet, which retires rather than removes the row:

ownsearch instances retire 1 --reason "replaced by another host"

Bring it back:

ownsearch instances restore 1

Apply the change to the fleet with a scan:

ownsearch scan --instance 1 --force

An instance is never deleted, so its probe history and events stay readable. retire is the delete step of the lifecycle, and restore returns a retired or missing instance to candidate. The same operations exist over MCP through ownsearch_db_crud, with the preview and one-use token flow.

AI-assisted database changes

Write mode is on by default. Every change still needs a preview, then a separate apply call that carries the one-use token, and OWNSEARCH_AI_DB_MODE=read-only refuses all of them.

1. The client calls ownsearch_db_crud with dry_run: true. OwnSearch validates the entity, the field allowlist, the value types, and the lifecycle transition, then returns a before-and-after preview and a one-use token with a lifetime. The minimum interval between applies is enforced when the token is used.
2. The client calls again with dry_run: false and the token. OwnSearch applies exactly the previewed change in one transaction, appends a lifecycle event, regenerates the snapshot, and returns the committed record.
3. To add an instance, call the same tool with operation: create, entity: instance, and url set to the instance address. The preview and token flow is identical, and the instance is stored exactly as ownsearch instances add stores it.
4. An expired token, a changed preview, a replayed token, a raw SQL field, an unknown field, or read-only mode all return a stable error with a remedy, and every refusal is written to the event history.

The tool can create or remove a policy override, update lifecycle and policy fields, retire an instance, or restore a retired or missing instance. It cannot delete an instance row, rewrite probes, or forge events. Setting status to active requires a passing hard gate.

Development

cargo fmt --all -- --check
cargo check --workspace --all-targets --all-features --locked
cargo clippy --workspace --all-targets --all-features --locked -- -D warnings
cargo test --workspace --all-features --locked
cargo test --workspace --all-features --locked --doc
RUSTDOCFLAGS="-D warnings" cargo doc --workspace --all-features --no-deps --locked
cargo llvm-cov --workspace --all-features --locked --fail-under-lines 80
cargo run -p ownsearch -- sbom
tools/verify.sh
tools/smoke.sh http://127.0.0.1:8774

The live test is ignored by default because it needs the public network:

cargo test -p ownsearch-core --test live -- --ignored --nocapture

Troubleshooting

• ownsearch.no_fleet: run ownsearch scan, then check ownsearch instances list.
• Every search returns no results and sources.failed says ownsearch.policy_refused: the selected instances deny JSON and HTML. Run from a network the instances accept, or check ownsearch instances list for quarantined records.
• ownsearch.state_corrupted: run ownsearch db verify and restore the newest backup.
• ownsearch.write_disabled: set ai_db_mode = "read-write".
• ownsearch.confirmation_expired: call the tool again with dry_run: true.
• Disk filling up: lower OWNSEARCH_KEEP_WINDOW for the ledger archives, event lines, and pre-migration backups, then run ownsearch db maintenance and read the warnings it prints. SQLite rows follow the fixed per-table windows listed under Local files and keep windows.
• A startup error names the setting and its bounds. Fix the value, or remove it to take the default.
• The process exits with code 75: it stopped making progress for OWNSEARCH_WATCHDOG_SECS. Check the logs for the last request, and let the supervisor start it again.

Privacy

Queries go to the public instances you selected and to the engines behind them. OwnSearch stores no raw query text. The ledger records a keyed hash of the arguments, the tool name, the status, the upstream HTTP status class, timing, byte counts, parser, and instance ids. There is no telemetry.

Security

• No unsafe code anywhere in the workspace.
• Discovery records are limited to public addresses, so a hostile document cannot point the scanner at a local or internal service.
• rustls with strict certificate validation on every outbound connection.
• Bounded response reads, connection limits, and request deadlines.
• Search results are data. Every string passes a sanitizer that removes invisible and control characters, URLs must be HTTP or HTTPS, and long values are cut with a visible marker.
• No secret is accepted on a command line, written to the profile, or printed by any command.
• The browser renders under a per-process profile with owner-only permissions and is closed when it hangs, goes idle, or outlives its maximum age. Only a browser OwnSearch launched is ever closed.

Report a vulnerability privately; SECURITY.md states how.

Support

Write to sazzad@devops.bd for a question, a bug report, or an install problem. Include the version, the command you ran, and the output. Security problems follow SECURITY.md, not this address's normal queue.

The runbook is docs/operations.md: service levels, detection, triage, self-healing behavior, and baselines.

Contributing

The repository has a single maintainer. Send a patch or a proposal to sazzad@devops.bd before starting large work, and run tools/verify.sh before sending a change. A change lands when the full gate passes; RELEASING.md lists what the gate runs.

Distribution

Each release carries sha256.sum, which covers every archive, the installers, the formula, and the MCPB bundle, and is signed with minisign. ownsearch sbom generates a CycloneDX 1.7 bill of materials on demand from Cargo.lock.

The release channels:

• GitHub release archives and checksums, signed with minisign.
• Shell and PowerShell installers.
• A Homebrew formula built from the release archives.
• The npm loader @devops-infinity/ownsearch.
• crates.io packages ownsearch-core and ownsearch.
• The MCP Registry entry io.github.devops-infinity/ownsearch.
• An MCP bundle (MCPB) for desktop clients.
• A ghcr.io/devops-infinity/ownsearch image.

A v* tag starts the release workflow, which builds the seven targets and then creates and publishes the GitHub release. The gate runs on the machine through tools/verify.sh before the tag exists, so the workflow repeats none of it. Every action in that workflow is pinned to a commit, so a moved tag cannot change what runs. RELEASING.md holds the ordered publish steps and the credentials each one needs.

License

MIT OR Apache-2.0, at your choice. The full texts are in LICENSE-MIT and LICENSE-APACHE. The HTML parser (scraper) brings MPL-2.0 components (cssparser, selectors, dtoa-short), used as unmodified dependencies; MPL-2.0 permits that use inside a larger work under another license.