Graylog: Choose input types
How to choose the right Graylog input type
👋 Welcome to the Stackhero documentation!
Stackhero offers a ready-to-use Graylog cloud solution that provides a host of benefits, including:
- Unlimited and dedicated SMTP email server included.
- Effortless updates with just a click.
- Customizable domain name secured with HTTPS (for example, https://logs.your-company.com).
- Optimal performance and robust security powered by a private and dedicated VM.
Save time and simplify your life: it only takes 5 minutes to try Stackhero's Graylog cloud hosting solution!
Graylog offers a variety of inputs. These inputs allow you to send logs to the Graylog server or enable Graylog to fetch logs from another source.
In almost every case, you will want to send data from an application or a device to the Graylog server. We will not cover the advanced cases where you want Graylog to fetch logs from elsewhere. However, if you wish to do this, rest assured, it is very straightforward.
To choose the best input type, you first need to decide on the log format you want to send to Graylog. We recommend using the GELF format, as detailed below.
Next, you will need to select the protocol: UDP or TCP.
Selecting the input format
The first decision is to choose the format of your log messages. There are three main options: RAW/Plaintext, Syslog, or GELF.
RAW/Plaintext format
This is the simplest format. It works with any system because it sends a full plaintext message. It is very easy to use and is perfect for initial tests, such as sending a message using the utility netcat.
To test it, you can create a "Raw/Plaintext UDP" input and then send a test log using one of the following command lines:
- From macOS:
echo "Hello Graylog from UDP" | nc -u -w1 -c <XXXXXX>.stackhero-network.com 5555 - From Linux:
echo "Hello Graylog from UDP" | nc -u -w1 <XXXXXX>.stackhero-network.com 5555
It is very easy, isn't it? However, because it is a basic format, you will soon encounter limitations. Therefore, we discourage using it for more than simple tests.
Syslog format
This is a well-known format used widely to store and send logs in server environments. You can find it in Linux, Windows, and *BSD, among other operating systems.
If you are using Syslog, Rsyslog, or another compatible service and wish to send logs from a server to Graylog, the syslog input is the simplest option.
However, since GELF is more powerful, if you can send GELF-formatted messages, we recommend doing so, as explained below.
Note that the syslog format is limited to 1024 characters.
GELF format (recommended)
GELF stands for "Graylog Extended Log Format". This format was created by Graylog and has become widely adopted across many products, establishing itself as a standard.
The GELF format is powerful because it sends logs in a structured way. Structured logging allows you to transmit multiple fields and values instead of a single unstructured text message. This makes it easier for Graylog to parse and retrieve the information.
For example, if you want to send a log that contains an IP address and a request type, you can send a message with two fields named ip and request_type. On the Graylog side, you will be able to search directly on these fields, such as searching for all requests with the query: request_type: POST.
GELF is based on JSON and must include at least the version, host, and short_message fields. You can then add additional fields, which must be preceded by an underscore (_).
For example, a simple message with an additional field named device_id looks like this:
{
"version": "1.1",
"host": "myIotDevice",
"short_message": "Something is happening",
"_device_id": "abcd"
}
You can then search for all messages from the device "abcd" on Graylog using the query: device_id: abcd.
For more details about GELF, you can refer to the official specification here: https://docs.graylog.org/en/4.0/pages/gelf.html.
If you want to send logs from servers or applications, you can use one of the many available libraries that automatically create and send your messages. Examples are available for Node.js, .NET, and Python.
Choosing the right protocol
Once you have selected the appropriate format (we hope you chose GELF!), you need to choose a communication protocol between TCP and UDP.
UDP protocol
The UDP protocol is quite simple. Your application sends the message to the server without requiring a confirmation of receipt. There is no dialogue between the client and the server, meaning the server does not confirm that it has received the message.
The advantage of this approach is that if your Graylog server is slow or down, it will not impact your application's performance. The downside is that you cannot be certain every log sent from your application will reach the server. Additionally, encryption is not available with UDP.
Pros
- Does not impact your application if Graylog is slow or down.
Cons
- Some messages may be lost.
- Encryption is not available.
- Message size is limited to 8192 bytes.
TCP protocol
The TCP protocol, in contrast to UDP, exchanges information between your application and Graylog. This means you receive a confirmation that the message has been received by Graylog.
The benefit of TCP is that you can resend a message if it is not received by Graylog (for instance, when the server is down). The drawback is that if Graylog is slow or unresponsive, your application must wait for a response or a timeout, potentially slowing it down.
Another benefit is that you can encrypt your communications with TLS (also known as SSL).
Pros
- Ensures that the message has been received.
- Encryption is available (TLS).
- No message size limitation
Cons
- Can slow down the sender if Graylog is slow to respond or down.
Conclusion
We hope this guide has provided you with the necessary information to choose the best format and protocol for sending your logs to Graylog.
Remember, you can run multiple inputs at the same time by using different ports for each. For example, you could have a "syslog UDP" input on port 514 alongside a "GELF TCP" input on port 12201.