imhttp: HTTP input module

Module Name:	imhttp
Author:	Nelson Yen

Purpose

Provides the ability to receive adhoc and plaintext syslog messages via HTTP. The format of messages accepted, depends on configuration. imhttp exposes the capabilities and the underlying options of the HTTP library used, which currently is civetweb.

Civetweb documentation:

Configuration Parameters

Note

Parameter names are case-insensitive.

Module Parameters

Ports

type	mandatory	format	default
string	no	ports	8080

Configures “listening_ports” in the civetweb library. This option may also be configured using the liboptions (below) however, this option will take precedence.

The ports parameter accepts a comma-separated list of ports. To bind to a specific interface, use the format IP:Port. For IPv6 addresses, enclose the IP in brackets, e.g., [::1]:8080.

Civetweb listening_ports

documentroot

type	mandatory	format	default
string	no	none	.

Configures “document_root” in the civetweb library. This option may also be configured using liboptions, however this option will take precedence.

Civetweb document_root

liboptions

type	mandatory	format	default
string	no	none	none

Configures civetweb library “Options”.

Civetweb Options

healthCheckPath

type	mandatory	format	default
string	no	path that begins with ‘/’	/healthz

Configures the request path for a simple HTTP health probe that returns 200 when the module is running. The endpoint is unauthenticated unless healthCheckBasicAuthFile or healthCheckApiKeyFile is set. Otherwise, bind the server to localhost or use CivetWeb access controls if external access is not desired.

Authentication Parameters

Parameter	Summary
healthCheckBasicAuthFile	Protects the imhttp health check endpoint with HTTP Basic Authentication.
healthCheckApiKeyFile	Protects the imhttp health check endpoint with API key authentication.
metricsBasicAuthFile	Protects the imhttp Prometheus metrics endpoint with HTTP Basic Authentication.
metricsApiKeyFile	Protects the imhttp Prometheus metrics endpoint with API key authentication.

metricsPath

type	mandatory	format	default
string	no	path that begins with ‘/’	/metrics

Exposes rsyslog statistics in Prometheus text format at the specified path. The endpoint emits counters such as imhttp_submitted_total and imhttp_failed_total. By default the endpoint does not enforce authentication, but it can be protected with metricsBasicAuthFile or metricsApiKeyFile. Leaving the default paths enabled creates user-visible URLs as soon as the module is loaded, so review firewall and access-control settings.

Authentication Parameters

Parameter	Summary
metricsBasicAuthFile	Protects the imhttp Prometheus metrics endpoint with HTTP Basic Authentication.
metricsApiKeyFile	Protects the imhttp Prometheus metrics endpoint with API key authentication.

Input Parameters

These parameters can be used with the “input()” statement. They apply to the input they are specified with.

Endpoint

type	mandatory	format	default
string	yes	path that begins with ‘/’	none

Sets a request path for an HTTP input. Path should always start with a ‘/’.

DisableLFDelimiter

type	mandatory	format	default
binary	no		off

By default LF is used to delimit msg frames, for data is sent in batches. Set this to ‘on’ if this behavior is not needed.

Name

type	mandatory	format	default
string	no		imhttp

Sets a name for the inputname property. If no name is set “imhttp” is used by default. Setting a name is not strictly necessary, but can be useful to apply filtering based on which input the message was received from.

Ruleset

type	mandatory	format	default
string	no		default ruleset

Binds specified ruleset to this input. If not set, the default ruleset is bound.

SupportOctetCountedFraming

type	mandatory	format	default
binary	no		off

Useful to send data using syslog style message framing, disabled by default. Message framing is described by RFC 6587 .

RateLimit.Interval

type	mandatory	format	default
integer	no	none	0

Specifies the rate-limiting interval in seconds. Set it to a number of seconds to activate rate-limiting.

RateLimit.Burst

type	mandatory	format	default
integer	no	none	10000

Specifies the rate-limiting burst in number of messages.

RateLimit.Name

Default: none

Type: string

Description:

Sets the name of the rate limit to use. This allows multiple listeners to share the same rate limiting configuration (and state). The name refers to a global rate limit object defined in the configuration.

Note: This parameter is mutually exclusive with ratelimit.interval and ratelimit.burst. If ratelimit.name is specified, local per-listener limits cannot be defined and any attempt to do so will result in an error (and the named rate limit will be used).

Added in version 8.2602.0.

flowControl

type	mandatory	format	default
binary	no	none	on

Flow control is used to throttle the sender if the receiver queue is near-full preserving some space for input that can not be throttled.

addmetadata

type	mandatory	format	default
binary	no	none	off

Enables metadata injection into $!metadata property. Currently, only header data is supported. The following metadata will be injected into the following properties:

$!metadata!httpheaders: HTTP header data will be injected here as key-value pairs. All header names will automatically be lowercased for case-insensitive access.
$!metadata!queryparams: query parameters from the HTTP request will be injected here as key-value pairs. All header names will automatically be lowercased for case-insensitive access.

Authentication Parameters

Parameter	Summary
basicAuthFile	Protects an imhttp input with HTTP Basic Authentication using an htpasswd file.
apiKeyFile	Protects an imhttp input with API key authentication using tokens listed in a file.

Statistic Counter

This plugin maintains global imhttp statistics. The statistic’s origin and name is “imhttp” and is accumulated for all inputs. The statistic has the following counters:

submitted - Total number of messages successfully submitted for processing since startup.
failed - Total number of messages failed since startup, due to processing a request.
discarded - Total number of messages discarded since startup, due to rate limiting or similar.

Prometheus Metrics

If metricspath is configured (default /metrics), imhttp exposes rsyslog statistics in the Prometheus text exposition format. The endpoint provides counters such as imhttp_submitted_total, imhttp_failed_total and imhttp_discarded_total.

The handler sends a Content-Length header and closes the connection. Proxies or load balancers must allow such responses. An imhttp_up gauge is exported alongside the full rsyslog statistics. Name collisions with other exporters are unlikely but should be documented in monitoring setups. To expose only the metrics endpoint, load the module without configuring any input() statements.

Error Messages

When a message is to long it will be truncated and an error will show the remaining length of the message and the beginning of it. It will be easier to comprehend the truncation.

Caveats/Known Bugs

module currently only a single HTTP instance, however multiple ports may be bound.

Examples

Example 1

This sets up an HTTP server instance on port 8080 with two inputs. One input path at ‘/postrequest’, and another at ‘/postrequest2’:

# ports=8080
# document root='.'
module(load="imhttp") # needs to be done just once

# Input using default LF delimited framing
# For example, the following HTTP request, with data body "Msg0001\nMsg0002\nMsg0003"
##
# - curl -si http://localhost:$IMHTTP_PORT/postrequest -d $'Msg0001\nMsg0002\nMsg0003'
##
# Results in the 3 message objects being submitted into rsyslog queues.
# - Message object with `msg` property set to `Msg0001`
# - Message object with `msg` property set to `Msg0002`
# - Message object with `msg` property set to `Msg0003`

input(type="imhttp"
      name="myinput1"
      endpoint="/postrequest"
      ruleset="postrequest_rs")

# define 2nd input path, using octet-counted framing,
# and routing to different ruleset
input(type="imhttp"
      name="myinput2"
      endpoint="/postrequest2"
      SupportOctetCountedFraming="on"
      ruleset="postrequest_rs")

# handle the messages in ruleset
ruleset(name="postrequest_rs") {
   action(type="omfile" file="/var/log/http_messages" template="myformat")
}

Example 2

This sets up an HTTP server instance on ports 80 and 443s (use ‘s’ to indicate ssl) with an input path at ‘/postrequest’:

# ports=8080, 443 (ssl)
# document root='.'
module(load="imhttp" ports="8080,443s")
input(type="imhttp"
      endpoint="/postrequest"
      ruleset="postrequest_rs")

Example 3

Bind imhttp to specific IP addresses (e.g., localhost only) to prevent external access. This example binds to IPv4 127.0.0.1 on port 8080 and IPv6 [::1] on port 8081.

module(load="imhttp" ports="127.0.0.1:8080, [::1]:8081")
input(type="imhttp" endpoint="/postrequest")

Example 4

imhttp can also support the underlying options of Civetweb using the liboptions option.

module(load="imhttp"
       liboptions=[
         "error_log_file=my_log_file_path",
         "access_log_file=my_http_access_log_path",
       ])

input(type="imhttp"
      endpoint="/postrequest"
      ruleset="postrequest_rs"
      )

Example 5

Expose a Prometheus metrics endpoint alongside an input path:

module(load="imhttp" ports="8080" metricspath="/metrics")
input(type="imhttp" endpoint="/postrequest")

# scrape statistics
# curl http://localhost:8080/metrics
# HELP imhttp_up Indicates if the imhttp module is operational (1 for up, 0 for down).
# TYPE imhttp_up gauge
imhttp_up 1
# HELP imhttp_submitted_total rsyslog stats: origin="imhttp" object="imhttp", counter="submitted"
# TYPE imhttp_submitted_total counter
imhttp_submitted_total 0

Example 6

Expose only a Prometheus metrics endpoint secured with Basic Authentication:

module(load="imhttp" ports="8080"
       metricspath="/metrics"
       metricsbasicauthfile="/etc/rsyslog/htpasswd")

# scrape statistics with credentials
# curl -u user:password http://localhost:8080/metrics

Example 7

Protect an input endpoint with API-key authentication:

module(load="imhttp" ports="8080")
input(type="imhttp"
      endpoint="/postrequest"
      apiKeyFile="/etc/rsyslog/imhttp-apikeys")

# send logs with an API key
# curl -H 'Authorization: ApiKey secret-token' \
#      -d 'message payload' http://localhost:8080/postrequest

Example 8

Allow either Basic or API-key authentication on the same input during a migration:

module(load="imhttp" ports="8080")
input(type="imhttp"
      endpoint="/postrequest"
      basicAuthFile="/etc/rsyslog/htpasswd"
      apiKeyFile="/etc/rsyslog/imhttp-apikeys")

Example 9

Protect the metrics endpoint with an API key:

module(load="imhttp" ports="8080"
       metricsPath="/metrics"
       metricsApiKeyFile="/etc/rsyslog/imhttp-apikeys")

# scrape statistics with an API key
# curl -H 'Authorization: ApiKey secret-token' http://localhost:8080/metrics

Support: rsyslog Assistant | GitHub Discussions | GitHub Issues: rsyslog source project

Contributing: Source & docs: rsyslog source project