Real-time Monitoring
Watch your audit happen live. WebSocket streaming gives you instant visibility into every URL crawled, every link found.
What Is Real-time Monitoring?
Real-time monitoring means you get instant updates as your website audit progresses—no waiting, no guessing, no black box. Using WebSocket technology (Socket.IO), our system streams live progress data directly to your browser as URLs are crawled.
The moment a page is fetched, you see it appear in the "FETCHING" list. When the fetch completes, counters update immediately: URLs fetched increments, newly discovered links are added to URLs found, external links are counted. All of this happens in real-time, with sub-second latency.
This isn't just a progress bar that estimates completion. This is actual, live data from the crawling engine. You see the exact URLs being processed, the current depth level, memory usage, elapsed time, and sitemap validation results—all updating continuously.
Real-time monitoring transforms the audit experience from "submit and wait" to "watch and understand." You can debug issues on the fly, cancel audits if they're going wrong, and share live progress with team members or clients.
See It In Action
Create an audit and watch the report page. You'll see live updates streaming in: URLs being fetched appear in real-time, progress bars move smoothly, and completion happens automatically with no page refresh needed.
Monitoring Features
Comprehensive real-time visibility into audit progress
Live bi-directional connection using Socket.IO. Get instant updates without polling or page refreshes.
See exactly which URLs are being fetched right now. Watch concurrent crawls happening in parallel.
Real-time counters for URLs fetched, URLs found, external links blocked, and current depth level.
Track crawlable vs non-crawlable sitemap URLs as they're discovered. No waiting until completion.
Live timer showing exactly how long the audit has been running. Updates every second.
Visual completion indicator based on page limits and depth. See how close you are to finishing.
What You Can Monitor
Every metric tracked and updated in real-time
OUTPUT_FILEReport filename being generated
Updates: Once at start
audit_1731536789.txtSITEMAP_AUDITCrawlable vs non-crawlable sitemap URLs
Updates: After each fetch
CAN_CRAWL(T): 245 / CAN_CRAWL(F): 12URLS_FETCHEDTotal pages successfully crawled
Updates: After each fetch
142URLS_FOUNDPages discovered but not yet crawled
Updates: After each fetch
387EXTERNAL_URLSExternal links found and blocked
Updates: After each fetch
52DEPTH_COUNTCurrent crawl depth level
Updates: When depth changes
Level 3FETCHINGURLs being fetched right now (concurrent)
Updates: Every fetch start/end
5 concurrent requestsMEMORYHeap used/total and RSS in MB
Updates: After each fetch
45.2 MB / 512 MB heap, 72 MB RSSELAPSED_TIMETime since audit started
Updates: Every second
00:03:47How It Works Under the Hood
The technical architecture enabling real-time streaming
Socket.IO
WebSocket communication layerReliable real-time messaging with automatic reconnection and fallback to polling
JSON Streaming
Structured progress data formatMachine-readable updates with consistent schema. Easy to parse and display.
stdout Unbuffering
Immediate output from CLI processUsing stdbuf -o0 disables buffering so updates stream instantly
Persistent State
Job survives disconnectsSQLite database maintains state. Reconnect anytime to resume watching.
Why Real-time Beats Other Approaches
See the difference that WebSocket streaming makes
User Experience
Black box. No visibility into progress.
Problems
Uncertainty, anxiety, can't debug
User Experience
Delayed updates. Server strain from constant requests.
Problems
Inefficient, not real-time
User Experience
Instant updates. Watch every URL being crawled.
Problems
None. Best approach.
Why It Matters
The business value of real-time visibility
Transparency
No black box waiting. You see exactly what's happening at every moment during the crawl.
100% visibilityDebugging
If something goes wrong, real-time logs reveal the issue immediately. Fix problems faster.
Instant feedbackTrust
Clients and stakeholders can watch audits in progress. Builds confidence in the process.
User confidenceControl
Cancel long-running audits if you see they're going off-track. No wasting resources.
Stop anytimeBuilding real-time streaming required solving multiple technical challenges:
- •Stdout buffering: Node.js buffers CLI output. We use stdbuf -o0 wrapper to disable buffering.
- •JSON parsing: Switched from console.log() to process.stdout.write() for reliable JSON formatting.
- •Field naming: Ensured JSON output matches frontend expectations (UPPERCASE format).
- •Reconnection: Socket.IO handles automatic reconnection if WebSocket drops.
- •State persistence: SQLite database ensures audit state survives disconnects and server restarts.
Experience Real-time Monitoring
See your audit come to life. Watch every URL being crawled, every metric updating, in real-time.
Free plan includes real-time monitoring • No credit card required