User-Agent Blocking 101: Protect APIs with SafeLine Rules

APIs are a favorite target for automated scripts, scrapers, and bots. Whether it’s someone hammering your endpoints with curl, or a bot written in Python, these requests can bypass normal traffic patterns and cause unnecessary load—or worse, expose vulnerabilities.

To combat this, SafeLine WAF introduces a User-Agent-based ruleset designed to detect and block common HTTP clients used in automation. It’s simple, effective, and especially useful if you want to keep your API endpoints human-friendly only.

What’s the Problem with Automated Calls?

Most programming languages ship with built-in or popular HTTP libraries that make API calls easy. By default, many of these clients send a User-Agent header that identifies the tool or language in use. Example:

• curl/7.88.0
• python-requests/2.31.0
• Go-http-client/1.1

While these libraries are great for developers, they can be exploited by automated tools to scan, scrape, or abuse APIs. If your API isn’t meant for public or automated consumption, blocking these calls can reduce risk and noise.

SafeLine’s User-Agent Ruleset: What’s Inside?

Version Compatibility: SafeLine 7.3.0 and above

How it works:

• Matches incoming requests based on known User-Agent patterns from common programming languages.
• Blocks requests that have no User-Agent (a common trait of some basic HTTP libraries).
• Stops tools before they can hit sensitive endpoints.

Blacklisted User-Agents include:

• JavaScript: superagent, axios

• Python: requests, urllib, http

• C/C++: curl, libcurl, libwww
• Go: net/http, httpie
• Java: okhttp
• Ruby: httpparty, open-uri, net/http
• C#: RestSharp

• Empty Header: No User-Agent present

Whitelist: None by default (you can customize).

Example SafeLine Configuration

Here’s what a simple ruleset looks like:

rules:
  - name: Block missing UA
    match: Header.User-Agent == null
    action: deny

  - name: Block curl and libwww
    match: Header.User-Agent matches "curl|libwww"
    action: deny

  - name: Block Python clients
    match: Header.User-Agent matches "requests|urllib|http"
    action: deny

  - name: Block Go clients
    match: Header.User-Agent matches "Go-http-client|httpie"
    action: deny

  - name: Block Ruby clients
    match: Header.User-Agent matches "ruby"
    action: deny

  - name: Block JavaScript HTTP libraries
    match: Header.User-Agent matches "superagent|axios"
    action: deny

  - name: Block C# clients
    match: Header.User-Agent matches "restsharp"
    action: deny

Why Use It?

• Stops unwanted automation: Keeps endpoints clean from scripts and bots.
• Protects lightweight APIs: Perfect for homelabs, self-hosted projects, or small business apps.
• Customizable: Add or remove patterns based on your environment.

Join the SafeLine Community

If you continue to experience issues, feel free to contact SafeLine support for further assistance.

• GitHub Repository
• Official Docs
• Discord Community