Shipping Ambassador Access Logs to Elasticsearch

Ambassador is one of the easiest and fastest way to setup an API gateway on Kubernetes. Ambassador uses Envoy internally which makes it highly stable, superb at performance and various configuration options. As this would become the first hit for all your requests, it is utmost important to log all the requests to Ambassador and have them accessible.

Elasticsearch is a great tool for viewing logs, searching and parsing them and also creating insightful dashboards out of them. We use Elasticsearch extensively for our log management. We had Ambassador pod logs being sent to Elasticsearch already, but we couldn't make any sense out of those logs because they also involved application startup and other info logs which was difficult to segregate and parse. We were mainly interested in having Access logs captured and parsed so that we can get much better insights out of it.

Ambassador Architecture

Ambassador Logservice Plugin #

Envoy exposes Access Log Service, Ambassador allows us to create a LogService plugin to use this functionality and get the access logs from Envoy in its standard format. Read this documentation for more information on Envoy's access log format fields.

By creating a LogService, you can configure Ambassador to report its access logs to a remote gRPC service. The remote access log service (or ALS) must implement the AccessLogService gRPC interface, defined in Envoy's als.proto.

We implemented a LogService plugin in Golang which listens on a gRPC server and implements the als.proto using Envoy's go-control-plane library. Additionally, in this plugin we restructure the logs to a more human readable structure, convert the metrics to standard units and finally ship them to Elasticsearch using Bulk Indexer for efficient and streaming bulk uploads. You can find the code in this Github repository. You can configure the Elasticsearch cluster endpoints, credentials and index name using environment variables.

Usage #

Easiest way to deploy Ambassador LogService for Elasticsearch on your Kubernetes setup is to use the deployment.yml file in the repo and apply it.

curl -o ambassador_logservice_es.yml https://raw.githubusercontent.com/krish512/ambassador_logservice_es/main/deployment.yml

Edit this ambassador_logservice_es.yml file and update value for the environment variable ELASTICSEARCH_ENDPOINTS to your Elasticseach cluster's endpoints. Update the namespace to be same where Ambassador is currently running. Finally, apply the deployment to your cluster.

kubectl apply -f ambassador_logservice_es.yml

You should see logservice pods running in Ambassador's namespace. This will create daily indices in Elasticsearch as ambassador-yyyy.MM.dd. You can refer to Ambassador LogService Documentation to configure for various parameters like ambassador_id, flush_interval_time, etc.

Next Steps #

With the parsed Ambassador access logs available in Elasticsearch, you may create dashboards over this data using Kibana to gain more insights. We have created a similar dashboard in Kibana which gives us broad metrics like Total number of 5xx, 4xx and 2xx response status codes sent in last 30 mins, the rate of requests per minute, rate of errors per minute and 95th percentile, 99th percentile and average of request/response latency

Ambassador Dashboard - Kibana

Conclusion #

Ambassador logs can be very helpful in evaluating your application's performance over time. It can play significant role in achieving serviceability in your ecosystem and ensure system reliability. Ambassador logs on Elasticsearch have proved very helpful for us as they alert us whenever applications cross their error budgets.

If you like this post you can share it on your favorite social media network.

For feedback, please ping me on Twitter.

Published