How do I import server logs?

Short answer

Create an API key with the Import logs scope (logs:write), then POST Nginx or Apache combined-format lines to /api/v1/server/logs/import as { "lines": [...] }. TapCub keeps only the lines it classifies as bots, so you can see crawlers that never run JavaScript, with status codes and response times. Human lines are dropped, so your visitor numbers do not change.

Why import logs

The page script only sees bots that execute JavaScript, and it has no status code. Many search and AI crawlers fetch HTML without running scripts. Your access log sees every request. After an import:

• Web analytics › Bots includes crawlers that never ran the script.
• Monitoring › Crawl health splits 4xx and 5xx out of the success rate; before an import, script records count as success because they carry no status.

1. Create a key

Under Configuration › Settings › API keys, create a key bound to the project with the Import logs scope. events:write is not enough, and a user-level key with no project is rejected. The project comes from the key; do not send a project id.

2. Check the line format

Only combined format is parsed: IP, two placeholders, a bracketed time, the request line, a three-digit status, bytes, a quoted referrer and a quoted user agent. An optional trailing duration in seconds becomes the response time in milliseconds.

203.0.113.8 - - [02/Oct/2026:08:01:02 +0800] "GET /robots.txt HTTP/1.1" 200 120 "-" "Googlebot/2.1" 0.012

Times use the offset in the line, and months must be English abbreviations.

3. Send a batch

Sample: send the last 2,000 lines of an Nginx log with jq and curl.

tail -n 2000 /var/log/nginx/access.log \
  | jq -R -s '{lines: split("\n") | map(select(length > 0))}' \
  | curl -X POST "https://api.pathclue.com/api/v1/server/logs/import" \
      -H "Authorization: Bearer wck_YOUR_KEY" \
      -H "Content-Type: application/json" \
      --data-binary @-

Limits: at most 5,000 lines per call, each at most 4,000 characters, and a 2 MB JSON body. Over the limit is rejected, not truncated. Run it on a schedule (for example every 10 minutes with cron) and keep track of what you have already sent so lines are not posted twice.

4. Read the response

{ "lines": 2000, "parsed": 1990, "accepted": 310, "skipped": 1680 }

In the sample response, 10 lines did not match the format and 1,680 were human or unclassified traffic.

Common problems

Related: how bot detection works, send server-side events for business events such as orders, and crawl errors.