Support for ElasticSearch as sink

Topics: Contributing, General
Feb 12 at 8:52 AM
We are using ElasticSearch (with Kibana as our Dashboard) as our log repository and wanted to use ETW and the BufferedEventPublisher class to do 'bulk' inserts. I have created a fork where we have implemented this. The work is perhaps 80% complete, missing unit tests and some more clean-up to do.

Would this be of interest to include in the release or is there another way we should do this? I see that it is possible to use "customSink", but as I see it this would make a nice contribution. What do you think?

(I made the error of creating an issue for this instead of coming here, please let me know if I should delete/remove/edit/something this)
Developer
Feb 14 at 1:48 PM
Thanks for posting -- your sink looks interesting!

My only question from a high level would be if SLAB itself should incorporate custom sinks (with external dependencies). In order to keep the dependencies clean, the sink would be a new project within source as well as being a new NuGet package.

Anyone have any thoughts?

~~
Randy Levy
entlib.support@live.com
Enterprise Library support engineer
Support How-to
Feb 14 at 1:59 PM
We only depend on Json.Net, no other external deps, since ES expects just json.

My main reason for making it a part of SLAB is because we rely on the BufferedEventPublisher class. If this was a public class we could just depend on the nuget package in a custom sink.
Developer
Feb 14 at 2:14 PM
Thanks for clarifying; somehow I thought NEST was involved.

Take a look at the Contributing Code page for the steps in submitting a contribution.


~~
Randy Levy
entlib.support@live.com
Enterprise Library support engineer
Support How-to
Feb 19 at 5:55 AM
The bigger thing here may be the limitations/issues in SLAB for extending it with new sinks. I would agree that SLAB itself should not need to incorporate custom sinks, but there are some enhancements and changes that could be made to better support sink development. One of those changes may be making BufferedEvenPublisher public or even better, changing things such that buffered events need not be implemented inside the sink itself and SLAB could handle that and simply feed a sink that supported batch a collection of events. The other change that would be necessary is the handling of configuration.

As for ElasticSearch sink in SLAB, it certainly makes more sense than table storage in my opinion.
Feb 19 at 7:22 AM
We would prefer to not contribute the ES sink to SLAB "proper", but deliver it as an independent nuget package.

Your suggestion of SLAB delivering a list of messages makes a lot of sense to me. That way SLAB would take care of bulking up messages securely and we would just put the messages in our store(s). Would be nice if somebody from the P&P team would chip in their opinions on this.
Coordinator
Feb 19 at 6:02 PM
Contributing to SLAB and packaging are two different aspects. You can still contribute to SLAB project. We, p&p, then perform a code review and additional test pass.
We can then package it in a separate NuGet package, just like the Windows Azure and the SQL Server sinks are released today.
Developer
Feb 19 at 6:15 PM
Edited Feb 19 at 6:33 PM
Hi, I believe we considered having a filter that would buffer events and then send as a bulk to the sink that would simply persist them in 1 go, but then we backed off that idea for a few reasons:

1) It would be much easier to do with a dependency to Rx, but we wanted to keep SLAB itself with as few dependencies as possible. If the user wished he could explicitly add a dependency on Rx and do extra filtering.
  • IMO this is not a big deal, and we could easily break this rule. The next 2 are the really important ones.
2) Each sink had different ways of figuring out how big the buffer windows were, and with dealing with retries.
  • So for example Windows Azure table sink would have a max of 1000 entries or also send after a few seconds even if it didn't reach that max (these are normal rules across all sinks), but would ALSO split the batches so that only entries with the same partition id would be batched together.
3) Retrying when there are transient faults would not be able to be optimized if we just got a batch from a previous observable and we wouldn't be able to regroup the batches.
  • If there are transient errors, there is no need to retry just that original batch, the sink could later try to send a bigger batch instead (so the original buffer that was sent to the sink if we were pre-filtering would not have a way to get bigger when retrying). Also synchronizing these longer running operations could be tricky, as there might be a lot of small batches waiting in a queue to be processed by the sink, and the sink might be stuck retrying.
What are your thoughts? Do you envision ways to overcome points 2 and 3 with a filter using Rx?

-Julian
Developer
Feb 19 at 6:41 PM
Edited Feb 19 at 6:42 PM
BTW, forgot to mention that I think it should be OK for the BufferedEventPublisher to become public to allow reuse by custom sinks.
I believe we kept it internal as we didn't do perf test on every possible use of the EventPublisher, so if others depend on it and use it in a different way from the 2 sinks that are currently using it, it might not perform as expected (the perf tests were done on the 2 sinks directly).
We just do not know, and there's no hint or warning signs that it wouldn't perform, it was just a precaution.
Feb 19 at 6:53 PM
Some great points.

With regard to #2.
To your point batch sizes are commonly set on number of items, and there are some other factors that one would want to consider with batch size for the sink, notably serialization size of the event object.

What about setting the batch size to either count and/or some arbitrary calculated value(size) using a callback, passing the entry returning a value? Batch size should not exceed x where x is either a provided callback that can evaluate the contents of <T> like the payload, or maybe a property of the event where a transform/conversion of the message is done prior to batching Size property or something.

The other interesting consideration with batch that comes up in the case of elasticsearch is that individual items of the batch can fail, although in those cases it's not transient and likely to fail the next time.

I don't have a good answer for #3 other than hoping and taking the potential performance hit in those cases. =) Need to sleep on that one.

Generally where #2 and #3 are going to be an issue then some more complex batching could be implemented in the sink as is done today.
Feb 20 at 10:24 AM
juliandominguez wrote:
BTW, forgot to mention that I think it should be OK for the BufferedEventPublisher to become public to allow reuse by custom sinks.
I believe we kept it internal as we didn't do perf test on every possible use of the EventPublisher, so if others depend on it and use it in a different way from the 2 sinks that are currently using it, it might not perform as expected (the perf tests were done on the 2 sinks directly).
We just do not know, and there's no hint or warning signs that it wouldn't perform, it was just a precaution.
We have done some basic perf testing with ES; looks very good. Dumps hundreds of messages a second, running both locally and in Azure Cloud Service. We could perhaps do some more perf testing. Is there anything like that setup already somewhere that we can reuse?
Feb 21 at 1:10 AM
I submitted a pull request with changes focused on custom sink development. This goes beyond simply moving configuration interfaces and more focused on custom sink development. The refactoring includes some changes to the azure table and sql sinks to make them behave more like a custom external sink.

1) Moved necessary or useful types for custom sinks from SemanticLogging.Etw to the core SemanticLogging project
2) Dependency changes
a) SemanticLogging.Etw no longer has a dependency on Azure libraries
b) SemanticLogging.Etw no longer has a dependency on Azure/SQL Sinks
3) Changed some access modifiers on types (Including BufferedEventPublisher)

Please have a look at the changes (comment and let me know if you would like to be added as a fork collaborator)
https://slab.codeplex.com/SourceControl/network/forks/trentmswanson/slabdev/
Feb 27 at 6:47 PM
ElasticSearch Sink Features - Thoughts?

There are a set of ElasticSearch features that have not been implemented. Would love some feedback and though on the features and potential changes to the sink.

Security
Mani brought up security, and although ES does not provide an AuthN mechanisms OOTB there are ES community plugins that could be utilized in addition to proxies for this reason. Some thoughts on how people are securing this today and what AuthN features would be important. I understand the importance here and our approach was to simply ACL the endpoint (firewall IP whitelist) or keep it internal and open. There are MANY other options that could certainly be implemented.

IndexName
The current implementation takes a configurable index name and appends -yyyy.MM.DD to it , placing event entries in an index by day. This is a pretty common practice and allows one to match ES templates for new index creation and easily search across indexes. Kibana default is logstash-[yyyy.MM.DD]. This is the approach that we used today.

Some thoughts on features for index naming

Allow for a handful of common formats and take Enum setting
Allow for this and/or fixed name index where nothing is appended?
A more complex, and flexible format string "logstash-{Environment[machinename]}-{EventEntry[timestamp]:yyyy.MM.DD}"?
Use OOTB and optionally take a callback for formatting?
Take an interface from configuration and provide some OOTB implementations? (IIndexName::GetIndexName( ... ) ? Similar to the Func

EventEntry Serialization
The current implementation flattens event entry payload and prepends non-configurable "Payload_" to this. This does not necessarily have to happen. I believe that ideally this behavior would be configuration "shouldFlattenPayload" along with prefix string.

The second item would be an ability to specify the field name for serialization. We could take a mapping as part of the configuration for the sink, allowing the consumer to specify EventDate should be serialized as "@timestamp" for example.

ES does allow for field_name set as part of a mapping.

UDP Support
Optionally support the ES UDP bulk endpoints