Log to ETW

[ppittle]

Is their support for logging to Etw?

I'm running NServiceBus on Azure Service Fabric, and ASF's preferred logging is Etw. Is there a LogFactory for Etw?

I was able to hack something together by creating a custom log4net appender and then using the NServiceBus.Log4Net factory, but this was less than ideal.

[SimonCropp]

@ppittle

No we dont have native integration for Etw at the moment.

So you have two options

1. write your own logger

This is fairly simple to do. There is a sample here http://docs.particular.net/samples/logging/custom-factory/ . or you can look at any of the full implementations https://github.com/Particular/NServiceBus.Log4Net https://github.com/Particular/NServiceBus.NLog https://github.com/Particular/NServiceBus.CommonLogging

2. Go through an existing log lib

So as you pointed out u tried Log4Net and it was possible but painful. Have you tried NLog with this extension https://github.com/NLog/NLog.Etw ?

Does that give you a path forward?

[ppittle]

@SimonCropp - I can keep the custom log4net appender I wrote and be alright. I just wanted to get this in front of your team as a feature request/recommendation. (Is there a better avenue than Issues to do this?)

[SimonCropp]

Is there a better avenue than Issues to do this

Nah an issue is fine for now.

I just wanted to get this in front of your team as a feature request/recommendation

Currently we have no plans to support it from a Particular company perspective. We try to have support for the most commonly use libs. hence log4net and nlog. It is difficult to get full coverage over all possible scenarios. the same goes for transports and persistences. Generally we support the community in building their own custom integrations. For example i really like Serilog so i build this https://github.com/SimonCropp/NServiceBus.Serilog on my own time.

If you want to spike a site project, with ETW support for NSB, I am more than happy to be involved.

[robv8r]

Following in Simon Cropp's footsteps, I've started the NServiceBus.EventSourceLogging project.

I haven't released a NuGet package for the project yet. It's far from complete and I'd like more input from the community before I set things in stone.

The initial version of the NServiceBus.EventSourceLogging (NSBETW) project writes to the Windows EventLog using the Operational and Debug EventChannels. The problem with this approach is that all events (log messages) for all instances of NServiceBus generated on a given server would log to the same Event Log. Clearly, this isn't ideal.

There are ways around this, such as providing an abstract class that did all the heavy lifting. It would be left to the consumer to create an EventSource that inherits from that abstract class and provide a different event source name. This could be automated using NuGet by including an example source file. However, NuGet 3.x has changed/deprecated the inclusion of source code files..

Also, there are some fundamental differences between "traditional" logging providers (such as Log4Net) and ETW. The biggest difference is "who" determines the current logging level. With providers such as Log4Net, a programmer would specify the logging level either in code or in a configuration file.

With ETW, the logging level isn't determined by the application. Instead, the logging level is determined by the consumer of the events. In this case, the consumer is the Windows Event Log service. However, most event sources (AKA: ETW providers) don't write to the Event Log. You can capture such events using PerfView, Microsoft Message Analyzer, logman.exe, or writing your own listener.

An EventSource simply emits events when a listener is present. When no listener is present, there's a good chance than no events are emitted (however, there's no guarantee of this.) The listener determines the logging level by telling the event source what it's listening for.

My best guess is that most people would want Critical (Fatal), Error, and Warn messages logged on the Operational Channel in the Event Log. Some people may want Informational messages on the Operational Channel as well. My opinion is that the events being logged aren't well structured enough to be logged to the Admin Channel.

I could continue writing to the Debug Channel, but that's never worked out well for me in the past. In most cases, once you enable the Debug or Analytic Channel, you have to restart the application to make it stop emitting Debug and Analytic events. In some cases, I had to restart the computer. Instead, I'd see Debug and Analytic (Trace) messages being consumed by one of the aforementioned official ETW listeners.

I'm open to input. Feel free to add it here or on the GitHub Repository for NServiceBus.EventSourceLogging project.

[SimonCropp]

@robv8r TBH u already seem to know significantly more that me about ETW. so not sure how much help i can be.

A few suggestions

• if you havnt already post an into to your project in the group https://groups.google.com/forum/#!forum/particularsoftware
• as soon as you have a working nuget send us a docs PR here http://docs.particular.net/samples/logging/ with a sample and a link here http://docs.particular.net/platform/extensions#logging. beta is fine

Sounds like your instinct on Debug Channel is correct.

From my perspective

• Critical (Fatal), Error, and Warn: make visible in production logs
• Info: make it optional to turn on in production
• Debug: only for developers

[andreasohlund]

Regarding the inclusion of source files in NuGets, we still use it for some of out NuGets and I wouldn't be surprised if there will be some kind of support for it in the future even if the NuGet team said it wouldn't be supported. So perhaps start down that path and keep your fingers crossed?

[robv8r]

The inclusion for "content files" for NuGet 3.3+ appears to be at compile time, not at design time. They still include the original content file approach for backward compatibility, but all signs are pointing to eventually dropping that feature. My guess is that backlash from the community would force their hand to continue supporting it for .Net Framework projects, but I doubt that they'd introduce the feature for .Net Core projects.

[robv8r]

I've got the code to a stable enough point that I feel comfortable releasing a beta version of the NServiceBus.EventSourceLogging ETW NuGet package.

I'm working on the documentation (http://docs.particular.net/samples/logging/) and would like some input from the NServiceBus team on how detailed the documentation should be. I can certainly follow in the footsteps of what's already there. However, I feel that ETW needs more explanation to those who aren't already familiar with it.

It's easy to go other logging frameworks' websites and read up on them. While ETW has quite a bit of information out there, most of it is like drinking from a fire hose. Microsoft's documentation caters primarily to the C++ crowd. Vance Morrison (an architect on the .NET Runtime Team, specializing in performance issues) has quite a few MSDN articles on the subject, but they can still leave your head spinning.

In addition to the standard usage documentation and samples, should I also provide more information about ETW? Should that documentation be part of the NServiceBus documentation or should I host it on the NServiceBus.EventSourceLogger GitHub site and link to it from the NServiceBus documentation?

The additional documentation might cover the following areas:

• What's an EventSource
• What's the role of an event producer
• What's the role of an event controller
• What's the role of an event consumer
• What's the difference between an in-process and out-of-process event consumer
• What's an event tracing session
• What tools are native to the Windows Operating system that work with ETW
• What other tools are available?
• Why use ETW instead of traditional logging frameworks
• How does ETW differ from traditional logging frameworks

[robv8r]

Another question. How do I test my changes to a forked and locally cloned version of docs.particular.net project? It looks similar to a jekyll site, but there's no _config.yml or index.html. (I'm not a jekyll expert by any means, but I'd expect to at least see a Gemfile.)

If it's served by some other means, can you give me a hint? Thanks.

[danielmarbach]

@robv8r the generation happens in docs engine which is not part of that repo. The docs engine is currently a private repo. That being said I don't think it is necessary to test it locally. The readme https://github.com/Particular/docs.particular.net/blob/master/README.md contains all the necessary information to be able to contribute. Adding content is normally a question of copy pasting an existing page (changing the metadata) and adding the necessary snippets to the snippets solution. Feel free to ping @Particular/docs-maintainers if you have specific questions or something is unclear. Btw. good stuff!

[danielmarbach]

However, I feel that ETW needs more explanation to those who aren't already familiar with it.

I think @Particular/docs-maintainers can shed more light on that. we usually try to avoid duplicating general information about a technology, framework etc. on our docs site. But based on your description it sounds that ETW docs are not really helpful. So in this case it might be beneficial to have a bit more information on the docs site. Another approach would be that you collect the information you gathered in your personal blog / website and then link from our docs site to your blog. Then your blog is the canonical truth about ETW (which gives you the rightfully deserved traffic and community momentum). Let's see what @Particular/docs-maintainers think

[robv8r]

@danielmarbach Thanks for both responses.

[SimonCropp]

Another approach would be that you collect the information you gathered in your personal blog / website and then link from our docs site to your blog. Then your blog is the canonical truth about ETW (which gives you the rightfully deserved traffic and community momentum).

This would be a better scenario than pushing it into our doco. We try not to become owners for documentation on software not directly related to NSB.

it sounds that ETW docs are not really helpful.

If this is the case then the owners of ETW should be pushed to improve their documentation, and then that can be referenced from the NSB docs site

Another question. How do I test my changes to a forked and locally cloned version of docs.particular.net project

The docs site is a custom app that we have chosen not to make public thus far. So far we havent had much of a requirement to do a local render of changes. It is mostly standard md. if u push to a PR then we have a build server that will verify the content for you and update the PR status.

Should that documentation be part of the NServiceBus documentation or should I host it on the NServiceBus.EventSourceLogger GitHub site and link to it from the NServiceBus documentation?

As i mentioned above I think for doco specific to ETW (not related specifically to your logger ) you should attempt to find a way to push that knowledge into the upstream ETW doco pages.

For doco on your logger. Thus far we have had a policy that NSB extensions not owned by particular should have their own hosted doco and we will only have a sample and pointers to that doco to help highlight the extension. For example we have a mongodb sample and links to the mongodb persister but the doco for the mongodb persister is hosted in its github repo.

However the @Particular/docs-maintainers have been discussing the viability of hosting doco for extensions to NSB. I ran this past @tekmaven a while ago but it slipped off the radar. So this would mean we would accept PRs for, and help with writing, doco for extensions to NSB. Is this something you think would be a good idea?

[robv8r]

I have no personal preference where the documentation lives. I'm happy to provide a documentation pull request (for docs.particular.net) with usage and samples that are similar to existing log providers. I'm also happy hosting what I feel is appropriate on github.io to expound on ETW and how the .Net EventSource type can be used across platforms.

As a side note, I intentionally named the logger the NServiceBus.EventSourceLogger because it's use isn't limited to just Event Tracing for Windows (ETW).

The EventSource type is fully supported in both the .Net Framework 4.6 and .Net Core.

For the .Net Framework 4.5 and above, the Microsoft.Diagnostics.Tracing.EventSource NuGet package provides everything that the .Net Framework 4.6 includes, and more. The NuGet package also includes Word Documents describing how to use the EventSource class and how the Event Registration process can be used to create a manifest that can be statically registered and used by off-the-shelf ETW consumers such as the Windows Event Log service.

I guess my point is that there's a lot of information out there (including official Microsoft documentation, NuGet Packages with embedded Word Documents, Vance Morrison's blog, etc.) The problem is that it's not in a format that would be easily digestible for the average .Net application developer.

I appreciate your input and will follow your recommendations of submitting usage and samples to the Particular documentation repo and linking to external documentation (whether it be official Microsoft docs or something I create on my own site) where appropriate.

Thanks

[robv8r]

I've released version 1.0.0 of the NServiceBus.EventSourceLogger to the official NuGet repository. As a convenience, I also uploaded a dependency only NuGet package named NServiceBus.ETW. I kept forgetting the name of my own NuGet package, so I couldn't expect others to find it.

In the somewhat near future, I'll update the docs. I just have to find some spare time.

[SimonCropp]

@robv8r i took the liberty of writing some documentation

• https://docs.particular.net/nservicebus/logging/eventsourcelogging
• https://docs.particular.net/samples/logging/eventsourcelogging/
• https://docs.particular.net/nservicebus/logging/#custom-logging

if i have anything wrong, or you would like to expand on any topics, a PR on the docs repo https://github.com/Particular/docs.particular.net would be greatly appreciated

@Particular/nservicebus-maintainers should we close this?

[andreasohlund]

@Particular/metrics-maintainers given the new metric support can this be closed?

[andreasohlund]

Closed based on the above comment