Custom SLAB Sink Development Changes

Topics: Contributing, General
Feb 19 at 6:07 PM
Discuss changes in SLAB to further simplify and improve the development and publishing of custom SLAB sinks.

1) Move SLAB configuration interfaces (ISinkElement and IFormatterElement) from “SemanticLogging.Etw” to “SemanticLogging”
This would allow a sink developer to support the out of process host advanced configuration as an option without having to take the dependency on SemanticLogging.Etw and simplify development and shipping a sink.

Note: This is a breaking change

2) Buffered Events - Either make BufferedEventPublisher public (allowing sink developers to use this) or the one I would much prefer and may be a lot more work, would be to work out an implementation that allows a sink to subscribe to a buffered IEnumerable<EventEntry> stream. This has the advantage that a sink would not need to handle the configuration of all the same setting types. I would picture this similar to SubscribeWithConversion, only SubscribeWithBufferedEvents(…)?

See This Discussion (https://slab.codeplex.com/discussions/531464)

---- Discussion Email Thread ----

From: Julian Dominguez []
Sent: Wednesday, February 19, 2014 9:52 AM
To: Trent Swanson; Grigori Melnik (PATTERNS & PRACTICES); Fernando Simonazzi (Clarius Consulting)
Cc: Christopher Bennage (PATTERNS & PRACTICES)
Subject: RE: SLAB Sink Question

Hey Trent, good suggestions. Do you mind if we copy this thread to codeplex and continue discussing there? This would allow others to chip in, and also be more transparent if we make the decision for a breaking change.
If so, do you want to do it or should I?

From: Trent Swanson []
Sent: Wednesday, February 19, 2014 7:11 AM
To: Julian Dominguez; Grigori Melnik (PATTERNS & PRACTICES); Fernando Simonazzi (Clarius Consulting)
Cc: Christopher Bennage (PATTERNS & PRACTICES)
Subject: RE: SLAB Sink Question

Gents,

The sink development is pretty easy, up to the point that you want to ship it.

What are your thoughts on the following.
1) Move SLAB configuration interfaces (ISinkElement and IFormatterElement) from “SemanticLogging.Etw” to “SemanticLogging”
This would allow a sink developer to support the out of process host advanced configuration as an option without having to take the dependency on SemanticLogging.Etw and simplify development and shipping a sink.

I discussed with Fernando and we had agreed this would be a good change. The issue is of course the fact that this will break existing users.

2) Either make BufferedEventPublisher public (allowing sink developers to use this) or the one I would much prefer and may be a lot more work, would be to work out an implementation that allows a sink to subscribe to a buffered IEnumerable<EventEntry> stream. This has the advantage that a sink would not need to handle the configuration of all the same setting types. I would picture this similar to SubscribeWithConversion, only SubscribeWithBufferedEvents(…)?

-Trent

From: Trent Swanson
Sent: Friday, February 14, 2014 4:21 PM
To: 'Julian Dominguez'; Grigori Melnik (PATTERNS & PRACTICES); Fernando Simonazzi (Clarius Consulting)
Cc: Christopher Bennage (PATTERNS & PRACTICES)
Subject: RE: SLAB Sink Question

Yeah, basically inverting the dependency. Ah.. yes, I see the point with having to take the dependency then on the out of process assemblies and the DLL explosion.

What if those interfaces were defined in “SemanticLogging”?

So, for someone implementing a custom sink and wanted to support out of process then I would have to take a dependency on “SemanticLogging.Etw?

-Trent

From: Julian Dominguez []
Sent: Friday, February 14, 2014 3:56 PM
To: Trent Swanson; Grigori Melnik (PATTERNS & PRACTICES); Fernando Simonazzi (Clarius Consulting)
Cc: Christopher Bennage (PATTERNS & PRACTICES)
Subject: RE: SLAB Sink Question

Not sure I understood. Do you mean inverting the assembly reference? That would make it so that if you just want to use the windows azure sink even in process, you have to take the entire dependency to out of process assembly, traceevent, etc.

I believe the only reason why the ETW project contains the WindowsAzureTableSinkElement is to avoid an explosion of DLLs by also requiring EmanticLogging.Etw.WindowsAzure for example.
For custom sinks (or external ones in that sense), you’ll probably need to create the 2 DLLs, unless you don’t mind having a single one that depends on the ETW stuff also.

From: Trent Swanson []
Sent: Friday, February 14, 2014 2:05 PM
To: Julian Dominguez; Grigori Melnik (PATTERNS & PRACTICES); Fernando Simonazzi (Clarius Consulting)
Cc: Christopher Bennage (PATTERNS & PRACTICES)
Subject: RE: SLAB Sink Question

I guess I could start these discussions up on codeplex and/or maybe test some of these changes and submit pull requests =)

Q: Shouldn’t the sink configuration implementation for the actual each add-on sink be the project for the sink?
For example:
Instead of SemanticLogging.Etw referencing SemanticLogging.WindowsAzure and containing WindowsAzureTableSinkElement.cs I would expect this code file to be in the WindowsAzure project implementing the ISink interface and referencing SemantiLogging.Etw.

It might make sense with the built-in console and rolling file appender but not the SQL and Azure Table. What do you guys think?

-Trent

From: Trent Swanson
Sent: Friday, February 14, 2014 11:19 AM
To: 'Julian Dominguez'; Grigori Melnik (PATTERNS & PRACTICES); Fernando Simonazzi (Clarius Consulting)
Cc: Christopher Bennage (PATTERNS & PRACTICES)
Subject: RE: SLAB Sink Question

Thanks Julian,

I’m still taking this in as I work here, and I guess I’m still struggling to see the value in using RX to transform the stream in this case, and how much decoupling the current implementations actually provide. I understand the intent and some potential value with the approach.

-Trent

From: Julian Dominguez []
Sent: Friday, February 14, 2014 9:58 AM
To: Trent Swanson; Grigori Melnik (PATTERNS & PRACTICES); Fernando Simonazzi (Clarius Consulting)
Cc: Christopher Bennage (PATTERNS & PRACTICES)
Subject: RE: SLAB Sink Question

This is an old decision, so I might be wrong, please correct me Fernando.

I think the idea was to decouple the observable stream from the observer (sink), so that you could use Rx to transform the original stream in however way you want, as long as the resulting stream can be observed by the sink. Additionally, you could be producing those events for the SQL sink to log in different ways, and not completely coupled to the one and only ObservableEventListener in SLAB.

-Julian

From: Trent Swanson []
Sent: Friday, February 14, 2014 9:49 AM
To: Grigori Melnik (PATTERNS & PRACTICES); Julian Dominguez; Fernando Simonazzi (Clarius Consulting)
Cc: Christopher Bennage (PATTERNS & PRACTICES)
Subject: SLAB Sink Question

Gents,

Can you provide some insight in the reason for the “EventRecord” type and the whole EventEntry >> EventRecord >> SqlDataRecord thing? It just seems unnecessary to me. I can ‘kind of’ understand it with the Azure one (CloudEventEntry.cs), but not with the SQL Sink. Even with the Azure one, what’s the purpose in exposing this type, and not simply using it within the sink, it would seem this is specific and internal to the azure table sink?

-Trent
Developer
Feb 19 at 8:12 PM
trentmswanson wrote:
1) Move SLAB configuration interfaces (ISinkElement and IFormatterElement) from “SemanticLogging.Etw” to “SemanticLogging”
This would allow a sink developer to support the out of process host advanced configuration as an option without having to take the dependency on SemanticLogging.Etw and simplify development and shipping a sink.

Note: This is a breaking change
I think this could be doable. The breaking change would require a recompilation of the current custom sinks, but there are not many out there yet, and private custom sinks can probably be rebuilt by users when they take in a new version of SLAB.

Additionally, there might be a way to keep the old interface in ETW, but does only inherit from the new interface in SLAB (and is also marked as obsolote), so perhaps this way there won't even be a need to rebuild the custom sinks? Maybe you can test this approach and send a pull request?
Feb 20 at 7:02 AM
trentmswanson wrote:
2) Buffered Events - Either make BufferedEventPublisher public (allowing sink developers to use this) or the one I would much prefer and may be a lot more work, would be to work out an implementation that allows a sink to subscribe to a buffered IEnumerable<EventEntry> stream. This has the advantage that a sink would not need to handle the configuration of all the same setting types. I would picture this similar to SubscribeWithConversion, only SubscribeWithBufferedEvents(…)?
I think a good first step (perhaps only) would be to make the BufferedEventPublisher public. That's a really easy fix, doesn't break anything and would basically allow us to do bulk logging without reinventing/copying the class.
Feb 21 at 12:13 AM
I meant to post what's on the elastic search sink discussion here.

Please see:
https://slab.codeplex.com/discussions/531464