Skip to content

1.0.0-Beta7

Pre-release
Pre-release
Compare
Choose a tag to compare
@andreasohlund andreasohlund released this 25 Mar 07:27
1.0.0-Beta7
e18b9cf

Introduction

ServiceControl is an auditing and monitoring service for NServiceBus endpoints and applications.

It gathers the audited messages forwarded by NServiceBus endpoints to the configured Audit queue and it exposes HTTP API that provides data and functionality services for ServiceInsight and ServicePulse

Prerequisites

  • .NET Framework 4.5 or later
    • (Note that this is a change from previous beta releases of ServiceControl which depended on .NET 4.0)

Important Note for previous Beta users of ServiceControl:

  • If you installed a previous beta release of ServiceControl (beta 6 or earlier) you will need to remove the embedded RavenDB database by running the following commands:
net stop "Particular ServiceControl"
RD C:\programdata\particular\ServiceControl\localhost-33333 /S
net start "Particular ServiceControl"
  • (Note the above command assumes you have installed ServiceControl using the default port 33333)
    • Once started, the ServiceControl will re-create a new RavenDB embedded database

Installation

  1. Run the ServiceControl installer and leave the default parameters as-is
    • ServiceControlPort: 33333
      • This is the default IP port ServiceControl will use to expose its HTTP API
    • Transport Type: MSMQ
      • Currently, only MSMQ is available for automatic installation and configuration, although ServiceControl does support all NServiceBus supported transports
      • For guidance on configuring ServiceControl to use transports other than MSMQ, please contact us
    • Instance Name: {free text}
    • Instance Description: {free text}
  2. If you change the default ServiceControl port, please make a note of the custom port you define, since this will be required for connecting ServiceInsight and ServicePulse to the installed instance of ServiceControl
  3. After accepting the license terms and conditions, click "Install" (installer will require elevated privileges)
  4. The installation process performs the following actions:
    1. Installs the Windows Service "Particular ServiceControl"
    2. Expose HTTP API for use by ServiceInsight and ServicePulse (default URI: http://localhost:33333/api)

Troubleshooting:

http://docs.particular.net/ServiceControl/troubleshooting

Issues fixed in this Beta

This release consist of these issues that were achieved through these commits.

Features

#230 Create a Chocolatey package

Note this is not dependent on MSMQ or DTC since ServiceControl can target other transports

#229 Facilitate the viewing of the actual message data in Service Insight for the Saga View.

For the saga view feature of ServiceIsnight, ServiceControl needs to be able to provide the message data in addition to the saga state changes. Facilitate any API changes that are needed on SC so that this data can be displayed in SI.

Why?
The benefit of being able to see all of the message data is significant. For example:
From a debugging standpoint a developer that is looking at the SagaView could troubleshoot the saga by being able to see the actual message and its property and the saga states to help determine that the problem was with a certain message that left the saga went out with a wrong data in the message.

#180 Add "Hosting Process" data to Heartbeats

As Opie, I need to be able to uniquely identify the specific endpoint hosting process that sent a heartbeat.
This is especially true when more than one instance of the endpoints is located in the same machine (AKA "side by side") It also applies when the endpoint is hosted in IIS or other custom hosting.

Visualization
  1. In SI, we will be including the "Hosting Process" information in endpoint properties / tooltip info.
  2. In SP, it will be displayed in the Endpoints List (monitored only, since it relies on heartbeats).
Proposed Implementation
  1. As discussed with @andreasohlund
    Content trimmed. See full issue

#116 Support expiration of audit messages

Use the expiration bundle to have raven auto expire audit messages.

Note: We're only expiring audit messages, not errors, heartbeats etc

Replaces Particular/ServicePulse#64

#106 Enrich audit messages returned with saga state information

To support data overlay for the message flow in ServiceInsight:

  • support multiple saga types per message
  • enrich initiating message with saga id + type (can be multiple sagas)
  • saga state change(s) caused by message (initiated/updated/complete)

Replaces: Particular/NServiceBus#1659

#105 Expose a http api to get saga state changes

Request:

GET /api/sagas/{saga-id}

Response

{
   "id":"93bfbbfe-f841-4ea5-a0fe-a2a800d8e08d",
   "saga_id":"93bfbbfe-f841-4ea5-a0fe-a2a800d8e08d",
   "saga_type":"MySaga",
   "changes":[
      {
         "start_time":"2014-01-03T12:09:37.5015257Z",
         "finish_time":"2014-01-03T12:09:37.5577757Z",
         "status":"new",
         "state_after_change":"[{\"$type\":\"MySagaData, ServiceControl.Sample\",\"Id\":\"93bfbbfe-f841-4ea5-a0fe-a2a800d8e08d\",\"Originator\":\"ServiceControl.Plugin.SagaAudit.Sample@ANDREAS\",\"OriginalMessageId\":\"124984c0-9742-460d-a6bc-a2a800d8ddd4\",\"SomeId\":\"00000000-0000-0000-0000-000000000000\"}]",
         "initiating_message":{
            "is_saga_timeout_message":false,
            "originating_endpoint":"ServiceControl.Plugin.SagaAudit.Sample",
            "originating_machine":"ANDREAS",
            "time_sent":"2014-01-03T12:09:35.284928Z",
            "message_type":"Message1"
         },
         "outgoing_messages":[
            {
               "destination":"ServiceControl.Plugin.SagaAudit.Sample@ANDREAS",
               "time_sent":"2014-01-03T12:09:37.54275Z",
               "message_type":"Message2"
            }
         ]
      }
   ]
}

#62 Capture saga state changes

Plugin to the new saga pipeline in 4.2 and capture state changes that then is reported back to the backend.

In the current spike:

https://github.com/Particular/NServiceBus/blob/sagapipeline/src/NServiceBus.Core/Pipeline/Behaviors/SagaStateAuditingBehavior.cs#L36

The following 3 data points are emitted to ServiceControl:
SagaInstanceCreated(sagaInstanceId,metadata, after));
SagaInstanceUpdated(sagaInstanceId,metadata, diff));
SagaInstanceCompleted(isagaInstanceId,metadata, diff));

Where diff and after is keyvalue pair containing the property name + value (converted to string)

Metadata is a list of key value pair containing:

  • MessageId
  • ConversationId
  • SagaType
  • DebugSessionId (Optional)

This feature needs to go in a separate plugin:

ServiceControl.Plugin.SagaStateChanges.dll

Improvements

#184 Robust message import steps

Default flow
  1. Big try/catch around the import code (ErrorQueueImport.Handle and AuditQueueImport.Handle)
  2. Write to database log the message that failed to be imported
  3. Write diagnostic log to disk
  4. Send email to OPS with path to log file
Critical flow
  1. Use circuit breaker to shutdown endpoint

#162 Customize embedded RavenDB drive/path

Current Status
  1. Currently, the embedded RavenDB is installed in C drive by default, with no option to change drive
  2. In many cases C drive is the system OS drive and is much smaller than other drives
  3. As a result, this sets a very hard artificial limit on the size of the SC database
Required Functionality
  1. Allow the user to specify in Config (post installation) the path of the database
  2. If the user wants to recreate the database from scratch - allow the user to specify a new or non-existant path
  3. If the user wants to move an existing database to a new path - allow the user to specify an existing path and copy the ravendb to that path
    Content trimmed. See full issue

#157 Support for large Message Body response limitation

  1. Related to Particular/ServiceInsight#185 (comment)
  2. Goal: allow SI to specify the length of message body it supports:
    • For display on screen
    • For "Save As" full download of message body
  3. Proposed behavior:
    1. SI will send a custom request header specifying "maximum accepted length"
    2. If message body is smaller than "maximum accepted length":
      • SC will return the body, with content-length header, and HTTP 200
    3. If message body is larger than "maximum accepted length":
      • SC will return HTTP 413 ("Request Entity Too Large") with message specifying the message body size
      • (content-length response header will include the actual response size, not message body size)
        Content trimmed. See full issue

#128 Messages relating to endpoints should contain the endpoint info inside the message

The message body should include the full intent of the message.

So in the case if ReportCustomCheckResult and EndpointHeartbeat the endpoint information should be contained within the message

#119 Provide Code snippet sample for using Heartbeats / Custom Checks with NServiceBus v3.x

#117 Make the /storage option available only by demand and off by default

Based on a discussion with @andreasohlund and @dannycohen,
Accessing SC data directly for example, http://localhost:33333/storage should only be available via a setting in SC.

By default, the option to access directly will be turned off.

#67 Document ServiceControl

http://stackoverflow.com/questions/19096354/nservicebus-serviceinsight-monitor-multiple-error-and-audit

some blurb from support issue:

  • It is recommended that you centralize you audit and error storage, so for your logical system you would have one error queue and one audit queue.
  • You can configure the error queue in your app.config file in MessageForwardingInCaseOfFaultConfig like so:

Or

At a machine level using the PowerShell comandlet:
Content trimmed. See full issue

#27 Author article on Heartbeat extensibility

See sample guidance in #24 (comment) and here: https://github.com/Particular/ServiceControl/blob/develop/Src/ServiceControl.EndpointPlugin/BusinessMonitoring/EndpointSLAHeartbeatProvider.cs

#20 Add port number to log location

The logging location needs to take into account the port number selected so that if more than one endpoint is installed in the same box, the logs are not placed into the same location.

This code needs to be updated https://github.com/Particular/ServiceControl/blob/master/Src/ServiceBus.Management/EndpointConfig.cs#L21

For the RavenDB path I'm using a combination of hostname + port, see https://github.com/Particular/ServiceControl/blob/master/Src/ServiceBus.Management/Settings.cs#L154

Bugs

#266 SC Indexing issues under load

#265 Fix the nuget min dependecies for the Plugin dlls

Currently for ServiceControl.Beta0007.1, the nuget min dependencies are set to 4.4.2.
While for SagaAudit it might be right, it needs to be set to 4.0.x for the heartbeat and custom checks so that it can be applied to exisiting 4.x endpoints.

Current min dependency:

<dependencies>
      <dependency id="NServiceBus" version="4.4.2.0" />
      <dependency id="NServiceBus.Interfaces" version="4.4.2.0" />
    </dependencies>

Content trimmed. See full issue

#264 When an endpoint implements a CustomCheck, a ComponentNotRegistered Exception is thrown in the endpoint

Steps to repro

  1. Install the latest SC from the release branch
  2. Add the latest custom checks plugin to a 4.4.2 endpoint
  3. Add the specified custom check.
  4. Start the endpoint
  5. Notice the following exception:
    image

NOTE: This does not happen for PeriodicCheck

CustomChecks included:
Content trimmed. See full issue

#262 SC installs and stops on WinServer2012

Repro
  1. installed latest SC build on Win2012 on Azure
  2. SC starts and then stops
  3. Log contents:
2014-03-07 10:03:15.2694|INFO|NServiceBus.Configure|Invocation of NServiceBus.IWantToRunBeforeConfiguration completed in 0.10 s
2014-03-07 10:03:15.6384|INFO|NServiceBus.Configure|Invocation of NServiceBus.Config.INeedInitialization completed in 0.00 s
2014-03-07 10:03:34.7700|INFO|System.RuntimeMethodHandle|Initialized expired document cleaner, will check for expired documents every 60 seconds
2014-03-07 10:03:34.7700|INFO|ServiceControl.Infrastructure.RavenDB.RavenBootstrapper|Index creation started

Content trimmed. See full issue

#261 Endpoints using Castle willl blow up if 2 plugins are installed

Repro
  1. New project
  2. install-package NServiceBus.Host
  3. install-package ServiceControl.Plugin.Heartbeat -pre
  4. install-package ServiceControl.Plugin.CustomChecks -pre
Exception

Content trimmed. See full issue

#249 FailedMessagePolicy saga is never marked as complete ?

@andreasohlund is there a reason why FailedMessagePolicy saga is never marked as complete ?

Also, if a user archives an error, should we delete the associated saga ?

#244 Other transport assemblies not being scanned

I have cloned ServiceControl and am on the develop branch

I've added RabbitMQ transport via ripple:

buildsupport\ripple.exe install -p ServiceControl NServiceBus.RabbitMQ

I modified the app.config to use RabbitMQ transport:

<appSettings>
    <add key="ServiceControl/TransportType" value="NServiceBus.RabbitMQ, NServiceBus.Transports.RabbitMQ" />

Content trimmed. See full issue

#241 Stop using the message id as the document id when storing processedmessages

Guids cause raven perf issues:

http://ayende.com/blog/162945/b-trees-and-why-i-love-them-part-i

We need to either generate a sequential id or let raven generate the id.

This will cause issues in the places where we rely on that id to do LoadDocument

#240 RavenDB needs to revalidate indexes in an async way

Otherwise it blocks

This revalidation happens if process is shutdown abnormally

#239 HeartbeatsComputation - Waited for 17,394ms for the query to return non stale result

This can happen when RavenDB has to revalidate its indexes

System.Reflection.TargetInvocationException: Exception has been thrown by the ta
rget of an invocation. ---> System.InvalidOperationException: Something went wro
ng when trying to satisfy one of the dependencies during composition, make sure
that you've registered all new dependencies in the container and inspect the inn
erexception for more details. ---> Autofac.Core.DependencyResolutionException: A
n exception was thrown while invoking the constructor 'Void .ctor(Nancy.INancyMo
duleCatalog, Nancy.INancyContextFactory, Nancy.Routing.IRouteSegmentExtractor, N
ancy.Routing.IRouteDescriptionProvider, Nancy.Culture.ICultureService)' on type

Content trimmed. See full issue

#225 Retry all with large number of messages not performant at all

I have just filled SC with 232,000 error messages and then issue a retry all via REST.

The code has been running for 20mins!

I don't think the current implementation works if there are a lot of errors to be retried.

We currently using the Stream api, what other options do we have?

//cc @andreasohlund @synhershko

#224 Shutdown not compeliting

In a busy system (audit queue not empty), if you issue a shutdown they system hangs.
Here is the log:

2014-02-12 11:02:48.8756|WARN|Raven.Database.Prefetching.PrefetchingBehavior|WASTE: Discarding future work item without using it, to reduce memory usage
2014-02-12 11:02:50.6251|WARN|Raven.Database.Prefetching.PrefetchingBehavior|WASTE: Discarding future work item without using it, to reduce memory usage
2014-02-12 11:03:01.4527|WARN|Raven.Database.Prefetching.PrefetchingBehavior|WASTE: Discarding future work item without using it, to reduce memory usage
2014-02-12 11:16:25.4779|INFO|NServiceBus.Unicast.UnicastBus|Initiating shutdown
.
2014-02-12 11:16:25.6490|INFO|ServiceBus.Management.Infrastructure.OWIN.OwinRunner|Api is now stopped
2014-02-12 11:16:25.6600|INFO|NServiceBus.Satellites.SatelliteLauncher|Stopped 4/4 ServiceControl.Operations.AuditQueueImport, ServiceControl,Version=0.8.0.0,Culture=neutral, PublicKeyToken=null satellite
2014-02-12 11:16:25.6600|INFO|NServiceBus.Satellites.SatelliteLauncher|Stopped 1/4 NServiceBus.Timeout.Hosting.Windows.TimeoutMessageProcessor, NServiceBus.Core, Version=4.5.0.0, Culture=neutral, PublicKeyToken=9fc386479f8a226c satellite

Content trimmed. See full issue

#214 [Heartbeat Plugin] Exception when working with ActiveMQ Transport

When running a solution where Endpoints have Heartbeat plugin configured, and ActiveMQ is configured as Transport, it throws the following exception.

NOTE: If Heartbeat plugin references are removed from endpoints, solution works just fine.

Message: The given key was not present in the dictionary.
StackTrace: at System.ThrowHelper.ThrowKeyNotFoundException()
at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
at NServiceBus.Transports.ActiveMQ.Encoders.TextMessageEncoder.Encode(TransportMessage message, ISession session) in c:\BuildAgent\work\d4de8921a0aabf04\src\ActiveMQ\NServiceBus.ActiveMQ\Encoders\TextMessageEncoder.cs:line 10
at NServiceBus.Transports.ActiveMQ.ActiveMqMessageEncoderPipeline.Encode(TransportMessage message, ISession session) in c:\BuildAgent\work\d4de8921a0aabf04\src\ActiveMQ\NServiceBus.ActiveMQ\ActiveMqMessageEncoderPipeline.cs:line 26
at NServiceBus.Transports.ActiveMQ.ActiveMqMessageMapper.CreateJmsMessage(TransportMessage message, ISession session) in c:\BuildAgent\work\d4de8921a0aabf04\src\ActiveMQ\NServiceBus.ActiveMQ\ActiveMqMessageMapper.cs:line 33
at NServiceBus.Transports.ActiveMQ.MessageProducer.SendMessage(TransportMessage message, String destination, String destinationPrefix) in c:\BuildAgent\work\d4de8921a0aabf04\src\ActiveMQ\NServiceBus.ActiveMQ\MessageProducer.cs:line 26
Content trimmed. See full issue

#213 Add an EventLogMappingDefinition for muting custom checks

#211 Add extra statuses to results returned from /api/messages

  1. Same message returns two different status, from different API's:
http://localhost:33333/API/messages

image

http://localhost:33333/API/errors

Content trimmed. See full issue

#209 SagaHistory document ends up with "Endpoint": null

Here is the document:

{
  "SagaId": "a97e89ca-5310-4195-a9ff-a2c20100dcc7",
  "SagaType": "MyServer.Saga.SimpleSaga",
  "Changes": [
    {
      "StartTime": "2014-01-29T04:35:12.4134001Z",
      "FinishTime": "2014-01-29T04:35:12.4684406Z",
      "Status": "New",
      "StateAfterChange": "[{\"$type\":\"MyServer.Saga.SimpleSagaData, MyServer\",\"Id\":\"a97e89ca-5310-4195-a9ff-a2c20100dcc7\",\"Originator\":\"MyServer@JOHNPC2\",\"OriginalMessageId\":\"ee488fec-bba6-4657-952c-a2c20100dca2\",\"OrderId\":\"069b1602-1e6b-4326-88f0-4c2db8b8de8b\"}]",

Content trimmed. See full issue

#206 After install, /storage returns HTTP 503

  1. Related to #198
  2. Repro:
    1. uninstall / re-install SC through installer
    2. Access /storage
    3. Note that the response is HTTP 503 (instead of 404)
  3. Seems like the installer adds the urlacl (see #198 (comment))

Tested on Particular.ServiceControl-1.0.0-Beta5

#199 Detected endpoint not reported in /endpoints

Repro
  1. Created a simple SM solution with a sending MVC and receiving host endpoints
  2. Sent a message, audited and properly displayed (see image below)
  3. However, only the receiving endpoint (Application2.Sales) is shown in Endpoints List (which correctly represents the output of /api/endpoints)
  4. The sending MVC endpoint (Application2.ECommerce) is not reported in Endpoints list

image

Tested with Particular.ServiceControl-1.0.0
Content trimmed. See full issue

#198 /storage returns HTTP 503 (server not available) - misleading

  1. Accessing /storage by default returns HTTP 503.
  2. I am assuming this is intentional, resulting from #117
  3. HTTP 503 is explained as: "The server is currently unavailable (because it is overloaded or down for maintenance)"
  4. It should return either:
    • 404 (with the goal of obfuscating the /storage existence)
    • or a 403 (Forbidden - i.e. the request was a valid request, but the server is refusing to respond to it. Unlike a 401 Unauthorized response, authenticating will make no difference)

Tested on Particular.ServiceControl-1.0.0-Beta5

// @indualagarsamy / @andreasohlund

#170 PeriodicCheck can cause system to hang on stop

Two issues with current impl.

  1. The current impl:
while (!cancellationToken.IsCancellationRequested)
{
    var result = p.PerformCheck();
    ReportToBackend(result, p.Id, p.Category, TimeSpan.FromTicks(p.Interval.Ticks*4));

    Thread.Sleep(p.Interval);
}

Content trimmed. See full issue

#167 Can't customize Name / Description properties

Repro
  1. In registry I updated the name / description values (see image below)
  2. Restarted SC service
  3. Same values appear in /api response
  4. Also tried adding config settings in ServiceControl.dll.config:
    <add key="ServiceControl/Name" value="DancoSC" />
    <add key="ServiceControl/Description" value="Danny's machine" />

Content trimmed. See full issue

#141 Exception in Periodic Check code is not reported

Repro
  1. Write Periodic Check
  2. Throw an exception in the Periodic Check code
  3. SP will not be notified of the exception, nor will it indicate that a custom check has failed (it will simply ignore the exception)
Expected behavior
  1. When a Customer Check throws an exception, treat it as a Custom Check failure
  2. In the CheckResult.Failed, include the basic exception info (type, message)

Content trimmed. See full issue

#92 Installer ignores service instance name and description

The settings specified in the installer for Name and Description:

image

Are not applies to SC instance:

{"audit_url":"http://localhost:33334/api/audit/{?page,per_page,direction,sort}",

...

"name":"ServiceControl",
"description":"Management support for NServiceBus"}

#60 Installer Transport selection display bugs

Several issues regarding the design implementation in #29:

  1. Can we implement the transport type dropdown to be in the same style as the textboxes ?
    • (alternatively, if we cannot implement a similar look, move it to a different page and implement it as a radio button group, with a radio option for each transport type)

image

  1. Note the length issue for the transport names:
    • (radio button group in a separate page may solve this as well ?)

image
Content trimmed. See full issue

Where to get it

You can download this release from:

Assets 3
Loading