This project is read-only.

Semantic Logging Application Block (SLAB) 1.1

Elasticsearch sink

This readme covers the Semantic Logging Application Block (SLAB) sink for publishing logs to an Elasticsearch Server 1.x (Note: the sink requires version 1.x and will not work with previous 0.xx versions of Elasticsearch). To download, install, and configure Elasticsearch on your platform of choice follow the instructions on the Elasticsearch site. When deploying and configuring Elasticsearch for logging it is recommended that the server and indexes are configured for logging workload. You can find more information and recommendations for configuring Elasticsearch server for logging here.

The sink can be used in– or out-of-process, and buffers events and writes them to elastic search server in batches through the _bulk service. The sink is configurable and in addition to the elastic search server endpoint URI other options can be configured:
  • instanceName - The name of the instance originating the entries that is either machine name or some other identifier.
  • connectionString - The Elasticsearch server URI (http:localhost:9200), for example.
  • Index - The index name prefix which is then formatted as index-{0:yyyy.MM.DD}.
  • Type - The Elasticsearch entry type.
  • flattenPayload - Flatten the payload collection when serializing event entries.
  • bufferInterval - The buffering interval to wait for events to accumulate before sending them to Microsoft Azure Storage.
  • bufferingCount - The buffering event entry count to wait before sending events to Elasticsearch.
  • maxBufferSize - The maximum number of entries that can be buffered while it's sending to Azure Storage before the sink starts dropping entries.
  • onCompletedTimeout - Defines a timeout interval for when flushing the entries after an OnCompleted call is received and before disposing the sink.

The following is a sample template that can be used to configure Elasticsearch logging indexes, and can be applied using either the CURL or SENSE tool which can be installed as a browser add-in. For more information on Elasticsearch templates and managing them see http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-templates.html.

Review and modify the following settings to meet your requirements. The template index matching "slab-*" should be updated to match that specified in creating the sink. The index shards and replicas should be tuned to match that of your deployed cluster and availability requirements. More information on these settings is available on the Elasticsearch site for configuring these settings.

{
    "template": "slab-*",
    "settings": {
      "index.refresh_interval" : "5s",
      "index.number_of_shards": 1,
      "index.number_of_replicas": 0,
      "index.query.default_field": "Message",
      "index.auto_expand_replicas": false
    },
    "mappings": {
        "etw": {
            "_all": { "enabled": false },
            "_source": { "compress": false },
            "dynamic_templates": [
                {
                    "payload_template" : {
                        "mapping": { "type": "string", "index": "not_analyzed" },
                        "match": "Payload.*"
                    }
                }
            ],
            "properties": {
              "EventId": {"type": "integer", "index": "not_analyzed" },
              "EventDate": {"type": "date", "index": "not_analyzed" },
              "Keywords": {"type": "long", "index": "not_analyzed" },
              "ProviderId": {"type": "string", "index": "not_analyzed" },
              "ProviderName": {"type": "string", "index": "not_analyzed" },
              "InstanceName": {"type": "string", "index": "not_analyzed" },
              "Level": {"type": "integer", "index": "not_analyzed" },
              "Message": {"type": "string", "analyzer": "whitespace" },
              "Opcode": {"type": "integer", "index": "not_analyzed" },
              "Task": {"type": "integer", "index": "not_analyzed" },
              "Version": {"type": "integer", "index": "not_analyzed" },
              "Payload": { "type": "object", "dynamic": true, "path": "full" },
              "ActivityId": { "type": "string", "index": "not_analyzed" },
              "RelatedActivityId": { "type": "string", "index": "not_analyzed" }
             }
        }
    }
}



In order to enable the Elasticsearch integration tests, you need to update the app.config file. The required change is to uncomment the app setting with the key ElasticsearchUrl and set the value to an Elasticsearch URL (the default is localhost on the default port).

Acknowledgements

Many thanks to Trent Swanson for contributing this guidance and helping drive the Elasticsearch sink work.

Last edited Mar 27, 2014 at 9:05 PM by gmelnik, version 4