Prometheus: Troubleshooting

Errors you may encounter with Prometheus

👋 Welcome to the Stackhero documentation!

Stackhero offers a ready-to-use Prometheus cloud solution that provides numerous benefits, including:

  • Alert Manager included to send alerts to Slack, Mattermost, PagerDuty, etc.
  • Dedicated email server to send unlimited email alerts.
  • Blackbox to probe HTTP, ICMP, TCP, and more.
  • Easy configuration with an online configuration file editor.
  • Effortless updates with just a click.
  • Optimal performance and robust security powered by a private and dedicated VM.

Save time and simplify your life: it only takes 5 minutes to try Stackhero's Prometheus cloud hosting solution!

Solving the error "received unsupported Content-Type "..." and no fallback_scrape_protocol specified for target"

With the release of Prometheus v3, it is now necessary for target servers to include a Content-Type header in their responses to inform Prometheus about the protocol of the metrics contained in the reply. This affects tools like Node Exporter, which should now respond to Prometheus HTTP requests with the appropriate Content-Type. For more details, you might want to check the Prometheus documentation.

If this requirement is not met, you might encounter an error message like this:

received unsupported Content-Type "application/octet-stream" and no fallback_scrape_protocol specified for target

Here are some ways you can resolve this issue:

1. Update your target servers

If you are working with a target server like Node Exporter, consider updating it to the latest version. Recent updates typically ensure that the Content-Type header is correctly defined, which should resolve the error in Prometheus.

2. Define the Content-Type header for your target servers

For custom target servers, such as an API route you have developed to return Prometheus metrics, you can directly set the Content-Type header in the response.

For example, if you are using HapiJS, instead of returning your metrics like this:

return metrics

You can set the Content-Type header like this:

return h.response(metrics).type('text/plain;version=0.0.4');

Here are the supported Content-Type headers based on the protocol of the metrics you are returning:

  • PrometheusProto: application/vnd.google.protobuf;proto=io.prometheus.client.MetricFamily;encoding=delimited
  • PrometheusText 0.0.4: text/plain;version=0.0.4
  • PrometheusText 1.0.0: text/plain;version=1.0.0;escaping=allow-utf-8
  • OpenMetricsText 0.0.1: application/openmetrics-text;version=0.0.1
  • OpenMetricsText 1.0.0: application/openmetrics-text;version=1.0.0

3. Define a fallback protocol

You can also define a fallback protocol in your prometheus.yml configuration file. This protocol will be used if the target server does not specify a Content-Type header.

Here is an example:

  - job_name: "my-job"
    # [...]
    fallback_scrape_protocol: PrometheusText0.0.4

Supported values are PrometheusProto, PrometheusText0.0.4, PrometheusText1.0.0, OpenMetricsText0.0.1, and OpenMetricsText1.0.0.

Other articles for Prometheus