Skip to content

Instantly share code, notes, and snippets.

@pchalamet

pchalamet/Silo.cs

Last active November 12, 2019 05:12
Show Gist options
  • Save pchalamet/d486e846a16cbf2b871712cd846fefa3 to your computer and use it in GitHub Desktop.
Save pchalamet/d486e846a16cbf2b871712cd846fefa3 to your computer and use it in GitHub Desktop.
Download ZIP
silo logs
Raw
docker-compose.yaml
version: '3'
services:
consul:
image: consul:latest
ports:
- "8500:8500"
command: "agent -server -bootstrap -client=0.0.0.0 -ui"
silo:
build:
dockerfile: Dockerfile
context: output/silo/
depends_on:
- consul
webapi:
build:
dockerfile: Dockerfile
context: output/webapi
ports:
- "5000:80"
depends_on:
- consul
- silo
Raw
Silo.cs
using System;
using System.Threading.Tasks;
using Microsoft.Extensions.Logging;
using Orleans;
using Orleans.Configuration;
using Orleans.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using cart.grain;
using System.Net;
using System.Linq;
using System.Net.NetworkInformation;
using System.Net.Sockets;
namespace silo
{
public class Program
{
public static int Main(string[] args)
{
return RunMainAsync().Result;
}
private static async Task<int> RunMainAsync()
{
try
{
var host = await StartSilo();
Console.WriteLine("Press ENTER to exit");
Console.ReadLine();
await host.StopAsync();
return 0;
}
catch (Exception ex)
{
Console.WriteLine(ex);
return 1;
}
}
private static async Task<ISiloHost> StartSilo()
{
// define the cluster configuration
var builder = new SiloHostBuilder()
.ConfigureEndpoints(siloPort: 11111, gatewayPort: 30000, listenOnAnyHostAddress: true)
.Configure<ClusterOptions>(options =>
{
options.ClusterId = "OrleansCluster";
options.ServiceId = "CartWebApi";
})
.AddMemoryGrainStorageAsDefault()
.UseConsulClustering(gatewayOptions =>
{
gatewayOptions.Address = new Uri("http://consul:8500/");
})
.ConfigureApplicationParts(parts => parts.AddApplicationPart(typeof(cart.grain.CartGrain).Assembly).WithReferences())
.ConfigureServices(svc => svc.AddSingleton<ICartService, CartService>())
.ConfigureLogging(logging => logging.AddConsole());
var host = builder.Build();
await host.StartAsync();
return host;
}
}
}
Raw
silo.dockerfile
FROM microsoft/dotnet:2.2-runtime
COPY . /app
WORKDIR /app
ENTRYPOINT ["dotnet", "silo.dll"]
Raw
silo.log
warn: Orleans.Runtime.NoOpHostEnvironmentStatistics[100708]
No implementation of IHostEnvironmentStatistics was found. Load shedding will not work yet
info: Orleans.Runtime.Scheduler.OrleansTaskScheduler[0]
Starting OrleansTaskScheduler with 4 Max Active application Threads and 2 system thread.
info: Orleans.Runtime.Silo[100404]
Silo starting with GC settings: ServerGC=False GCLatencyMode=Interactive
warn: Orleans.Runtime.Silo[100405]
Note: Silo not running with ServerGC turned on - recommend checking app config : <configuration>-<runtime>-<gcServer enabled="true">
warn: Orleans.Runtime.Silo[100405]
Note: ServerGC only kicks in on multi-core systems (settings enabling ServerGC have no effect on single-core machines).
info: Orleans.Runtime.Silo[100403]
-------------- Initializing silo on host 0091df5ace4f MachineName 0091df5ace4f at 172.19.0.3:11111, gen 291747614 --------------
info: Orleans.Runtime.Silo[100415]
Starting silo Silo_787b0
info: Orleans.Runtime.Messaging.IncomingMessageAcceptor[101010]
Opened a listening socket at address 0.0.0.0:11111
info: Orleans.Runtime.Messaging.IncomingMessageAcceptor[101010]
Opened a listening socket at address 0.0.0.0:30000
info: Orleans.Runtime.ConsistentRing.VirtualBucketsRingProvider[0]
Starting VirtualBucketsRingProvider on silo S172.19.0.3:11111:291747614/x762F81A6.
info: Orleans.Runtime.ConsistentRing.VirtualBucketsRingProvider[103003]
Added Server S172.19.0.3:11111:291747614/x762F81A6. Current view: [S172.19.0.3:11111:291747614 -> <MultiRange: Size=x100000000, %Ring=100.000%>]
info: Orleans.Runtime.ConsistentRing.VirtualBucketsRingProvider[103005]
-NotifyLocalRangeSubscribers about old <(0 0], Size=x100000000, %Ring=100.000%> new <MultiRange: Size=x100000000, %Ring=100.000%> increased? True
info: Orleans.Runtime.Silo[0]
Skip multicluster oracle creation (no multicluster network configured)
info: Orleans.Runtime.Silo[100422]
-------------- Started silo S172.19.0.3:11111:291747614, ConsistentHashCode 762F81A6 --------------
info: Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.SystemQueue[0]
Starting AsyncAgent Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.SystemQueue on managed thread 1
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.SystemQueue0 on managed thread 8
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.SystemQueue1 on managed thread 9
info: Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue[0]
Starting AsyncAgent Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue on managed thread 1
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue0 on managed thread 10
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue1 on managed thread 11
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue2 on managed thread 12
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue3 on managed thread 13
info: Orleans.Runtime.Silo[100452]
Start Scheduler took 20 Milliseconds to finish
info: Orleans.Runtime.SiloLifecycleSubject[100453]
Stage -2147483648: Orleans.Runtime.SiloOptionsLogger
info: Orleans.Runtime.SiloLifecycleSubject[100453]
Stage 2000: Orleans.Runtime.InsideRuntimeClient, Orleans.Runtime.Silo
info: Orleans.Runtime.SiloLifecycleSubject[100453]
Stage 4000: Orleans.Runtime.Silo
info: Orleans.Runtime.SiloLifecycleSubject[100453]
Stage 8000: HostedClient, Orleans.Runtime.Silo
info: Orleans.Runtime.SiloLifecycleSubject[100453]
Stage 10000: Orleans.Runtime.Versions.GrainVersionStore
info: Orleans.Runtime.SiloLifecycleSubject[100453]
Stage 19999: Orleans.Runtime.Silo
info: Orleans.Runtime.SiloLifecycleSubject[100453]
Stage 20000: Orleans.Runtime.Silo
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.ActivationCountBasedPlacementOptions:
ChooseOutOf: 2
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.ClusterMembershipOptions:
NumMissedTableIAmAliveLimit: 2
LivenessEnabled: True
ProbeTimeout: 00:00:10
TableRefreshTimeout: 00:01:00
DeathVoteExpirationTimeout: 00:02:00
IAmAliveTablePublishTimeout: 00:05:00
MaxJoinAttemptTime: 00:05:00
ExpectedClusterSize: 20
ValidateInitialConnectivity: True
UseLivenessGossip: True
NumProbedSilos: 3
NumMissedProbesLimit: 3
NumVotesForDeathDeclaration: 2
DefunctSiloExpiration: 7.00:00:00
DefunctSiloCleanupPeriod:
IsRunningAsUnitTest: False
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.ConsistentRingOptions:
NumVirtualBucketsConsistentRing: 30
UseVirtualBucketsConsistentRing: True
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.EndpointOptions:
AdvertisedIPAddress: 172.19.0.3
SiloPort: 11111
GatewayPort: 30000
SiloListeningEndpoint: 0.0.0.0:11111
GatewayListeningEndpoint: 0.0.0.0:30000
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.GrainCollectionOptions:
CollectionQuantum: 00:01:00
CollectionAge: 02:00:00
DeactivationTimeout: 00:00:30
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.GrainDirectoryOptions:
CachingStrategy: Adaptive
CacheSize: 1000000
InitialCacheTTL: 00:00:30
MaximumCacheTTL: 00:04:00
CacheTTLExtensionFactor: 2
LazyDeregistrationDelay: 00:01:00
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.GrainVersioningOptions:
DefaultCompatibilityStrategy: BackwardCompatible
DefaultVersionSelectorStrategy: AllCompatibleVersions
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.LoadSheddingOptions:
LoadSheddingEnabled: False
LoadSheddingLimit: 95
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.MemoryGrainStorageOptions-Default:
NumStorageGrains: 10
InitStage: 10000
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.MultiClusterOptions:
HasMultiClusterNetwork: False
MaxMultiClusterGateways: 10
BackgroundGossipInterval: 00:00:30
UseGlobalSingleInstanceByDefault: True
GlobalSingleInstanceNumberRetries: 10
GlobalSingleInstanceRetryInterval: 00:00:30
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.NetworkingOptions:
OpenConnectionTimeout: 00:00:05
MaxSocketAge: 10675199.02:48:05.4775807
MaxSockets: 500
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.PerformanceTuningOptions:
DefaultConnectionLimit: 200
Expect100Continue: False
UseNagleAlgorithm: False
MinDotNetThreadPoolSize: 200
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.ProcessExitHandlingOptions:
FastKillOnProcessExit: True
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.SchedulingOptions:
PerformDeadlockDetection: False
AllowCallChainReentrancy: True
MaxActiveThreads: 4
DelayWarningThreshold: 00:00:10
ActivationSchedulingQuantum: 00:00:00.1000000
TurnWarningLengthThreshold: 00:00:00.2000000
MaxPendingWorkItemsSoftLimit: 0
EnableWorkerThreadInjection: False
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.SerializationProviderOptions:
FallbackSerializationProvider:
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.SiloMessagingOptions:
SiloSenderQueues: 2
GatewaySenderQueues: 2
MaxForwardCount: 2
ClientDropTimeout: 00:01:00
ClientRegistrationRefresh: 00:05:00
MaxEnqueuedRequestsSoftLimit: 0
MaxEnqueuedRequestsHardLimit: 0
MaxEnqueuedRequestsSoftLimit_StatelessWorker: 0
MaxEnqueuedRequestsHardLimit_StatelessWorker: 0
MaxRequestProcessingTime: 02:00:00
AssumeHomogenousSilosForTesting: False
DEFAULT_SHUTDOWN_REROUTE_TIMEOUT: 00:00:10
ShutdownRerouteTimeout: 00:00:10
ResponseTimeout: 00:00:30
ResponseTimeoutWithDebugger: 00:30:00
MaxResendCount: 0
ResendOnTimeout: False
DropExpiredMessages: True
BufferPoolBufferSize: 4096
BufferPoolMaxSize: 10000
BufferPoolPreallocationSize: 250
PropagateActivityId: False
LargeMessageWarningThreshold: 85000
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.SiloOptions:
SiloName: Silo_787b0
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.StatisticsOptions:
PerfCountersWriteInterval: 00:00:30
LogWriteInterval: 00:05:00
CollectionLevel: Info
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.TelemetryOptions:
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.TypeManagementOptions:
TypeMapRefreshInterval: 00:01:00
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.SiloOptionsLogger started in stage -2147483648 which took 16 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Starting lifecycle stage -2147483648 took 19.8566 Milliseconds
info: Orleans.Runtime.Silo[100401]
Silo Start()
info: Orleans.Runtime.Silo[100430]
Configured ThreadPool.SetMinThreads() to values: 200,200. Previous values are: 2,2.
info: Orleans.Runtime.Silo[100436]
Configured .NET ServicePointManager to Expect100Continue=False, DefaultConnectionLimit=200, UseNagleAlgorithm=False to improve Azure storage performance.
info: Orleans.Runtime.Silo[100452]
ConfigureThreadPoolAndServicePointSettings took 1 Milliseconds to finish
info: Orleans.Runtime.GrainTypeManager[101711]
Loaded grain type summary for 8 types:
Grain class cart.grain.cart.grain.CartGrain [-1141067427 (0xBBFCB15D)] from cart.grain.dll implementing interfaces: cart.grain.ICart [1651422567 (0x626EB567)]
Grain class Orleans.Providers.Orleans.Providers.MemoryStreamQueueGrain [1359359215 (0x51062CEF)] from OrleansProviders.dll implementing interfaces: Orleans.Providers.IMemoryStreamQueueGrain [1158152105 (0x4507FFA9)]
Grain class Orleans.Runtime.Development.Orleans.Runtime.Development.DevelopmentLeaseProviderGrain [940524075 (0x380F422B)] from Orleans.Runtime.dll implementing interfaces: Orleans.Runtime.Development.IDevelopmentLeaseProviderGrain [-1234611069 (0xB6695483)]
Grain class Orleans.Runtime.Management.Orleans.Runtime.Management.ManagementGrain [1954798034 (0x7483D9D2)] from Orleans.Runtime.dll implementing interfaces: Orleans.Runtime.IManagementGrain [-1734666656 (0x989B1660)]
Grain class Orleans.Runtime.ReminderService.Orleans.Runtime.ReminderService.GrainBasedReminderTable [-55315191 (0xFCB3F509)] from Orleans.Runtime.dll implementing interfaces: Orleans.IReminderTableGrain [-1135060418 (0xBC585A3E)]
Grain class Orleans.Runtime.Versions.Orleans.Runtime.Versions.VersionStoreGrain [331003096 (0x13BAB4D8)] from Orleans.Runtime.dll implementing interfaces: Orleans.Runtime.Versions.IVersionStoreGrain [-297600501 (0xEE42FA0B)], Orleans.IGrainWithStringKey [-1277021679 (0xB3E23211)]
Grain class Orleans.Storage.Orleans.Storage.MemoryStorageGrain [819348208 (0x30D642F0)] from OrleansProviders.dll implementing interfaces: Orleans.Storage.IMemoryStorageGrain [577125491 (0x22663C73)]
Grain class Orleans.Streams.Orleans.Streams.PubSubRendezvousGrain [1903070868 (0x716E8E94)] from Orleans.Runtime.dll implementing interfaces: Orleans.Streams.IPubSubRendezvousGrain [1746702088 (0x681C8F08)]
info: Orleans.Runtime.InsideRuntimeClient[100452]
Start InsideRuntimeClient took 9 Milliseconds
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.InsideRuntimeClient started in stage 2000 which took 11 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.Silo started in stage 2000 which took 18 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Starting lifecycle stage 2000 took 21.1624 Milliseconds
info: Orleans.Runtime.Messaging.IncomingMessageAcceptor[0]
Starting AsyncAgent Runtime.Messaging.IncomingMessageAcceptor on managed thread 7
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Messaging.IncomingMessageAcceptor0 on managed thread 14
info: Runtime.Messaging.SiloMessageSender/PingSender[0]
Starting AsyncAgent Runtime.Messaging.SiloMessageSender/PingSender on managed thread 7
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Messaging.SiloMessageSender/PingSender0 on managed thread 15
info: Runtime.Messaging.SiloMessageSender/SystemSender[0]
Starting AsyncAgent Runtime.Messaging.SiloMessageSender/SystemSender on managed thread 7
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Messaging.SiloMessageSender/SystemSender0 on managed thread 16
info: Orleans.Runtime.Silo[100452]
Start Message center took 11 Milliseconds to finish
info: Runtime.Messaging.IncomingMessageAgent/Ping[0]
Starting AsyncAgent Runtime.Messaging.IncomingMessageAgent/Ping on managed thread 7
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Messaging.IncomingMessageAgent/Ping0 on managed thread 17
info: Runtime.Messaging.IncomingMessageAgent/System[0]
Starting AsyncAgent Runtime.Messaging.IncomingMessageAgent/System on managed thread 7
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Messaging.IncomingMessageAgent/System0 on managed thread 18
info: Runtime.Messaging.IncomingMessageAgent/Application[0]
Starting AsyncAgent Runtime.Messaging.IncomingMessageAgent/Application on managed thread 7
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Messaging.IncomingMessageAgent/Application0 on managed thread 19
info: Orleans.Runtime.Silo[100452]
Start Incoming message agents took 32 Milliseconds to finish
info: Orleans.Runtime.GrainDirectory.LocalGrainDirectory[0]
Start
info: Runtime.GrainDirectory.AdaptiveDirectoryCacheMaintainer`1[0]
Starting AsyncAgent Runtime.GrainDirectory.AdaptiveDirectoryCacheMaintainer`1 on managed thread 7
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.GrainDirectory.AdaptiveDirectoryCacheMaintainer`10 on managed thread 21
info: Runtime.GrainDirectory.GlobalSingleInstanceActivationMaintainer[0]
Starting AsyncAgent Runtime.GrainDirectory.GlobalSingleInstanceActivationMaintainer on managed thread 7
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.GrainDirectory.GlobalSingleInstanceActivationMaintainer0 on managed thread 22
info: Orleans.Runtime.Silo[100452]
Start local grain directory took 55 Milliseconds to finish
info: Orleans.Runtime.Silo[100452]
Init implicit stream subscribe table took 9 Milliseconds to finish
info: Orleans.Runtime.Silo[100452]
Create system targets and inject dependencies took 44 Milliseconds to finish
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.Silo started in stage 4000 which took 166 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Starting lifecycle stage 4000 took 166.3864 Milliseconds
info: Orleans.Runtime.Catalog[100507]
Before collection#1: memory=7MB, #activations=0, collector=<#Activations=0, #Buckets=0, buckets=[]>.
info: Orleans.Runtime.Catalog[100508]
After collection#1: memory=8MB, #activations=0, collected 0 activations, collector=<#Activations=0, #Buckets=0, buckets=[]>, collection time=00:00:00.0198913.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer HostedClient started in stage 8000 which took 46 Milliseconds.
info: Orleans.Runtime.Silo[100452]
Init grain services took 4 Milliseconds to finish
info: Orleans.Runtime.MembershipService.MembershipOracleData[100603]
MembershipOracle starting on host = 0091df5ace4f address = S172.19.0.3:11111:291747614 at 2019-03-31 17:00:14.539 GMT, backOffMax = 00:00:20
info: Orleans.Runtime.MembershipService.MembershipOracleData[100652]
-CleanupTable called on silo startup. Membership table 3 silos, 0 are Active, 3 are Dead, Version=<0, 0>. All silos: [SiloAddress=S172.19.0.3:11111:291747572 SiloName=Silo_efcb6 Status=Dead, SiloAddress=S172.19.0.3:11111:291745251 SiloName=Silo_4aeb2 Status=Dead, SiloAddress=S172.19.0.3:11111:291745222 SiloName=Silo_39b85 Status=Dead]
info: Orleans.Runtime.MembershipService.MembershipOracleData[100660]
Starting IAmAliveUpdateTimer.
info: Orleans.Runtime.Silo[100452]
Starting local silo status oracle took 9314 Milliseconds to finish
info: Orleans.Runtime.Silo[100452]
Init type manager took 12 Milliseconds to finish
info: Orleans.Runtime.CounterStatistic[100702]
Starting Windows perf counter stats collection with frequency=00:00:30
info: Orleans.Runtime.Silo[100452]
Start silo statistics took 1 Milliseconds to finish
info: Orleans.Runtime.DeploymentLoadPublisher[0]
Starting DeploymentLoadPublisher.
info: Orleans.Runtime.DeploymentLoadPublisher[0]
Started DeploymentLoadPublisher.
info: Orleans.Runtime.Silo[100452]
Start deployment load collector took 17 Milliseconds to finish
info: Orleans.Runtime.Watchdog[0]
Starting Silo Watchdog.
info: Runtime.Watchdog[0]
Starting AsyncAgent Runtime.Watchdog on managed thread 24
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.Silo started in stage 8000 which took 9363 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Starting lifecycle stage 8000 took 9410.6872 Milliseconds
info: Orleans.Storage.MemoryGrainStorage.Default[0]
Init: Name=Default NumStorageGrains=10
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.Versions.GrainVersionStore started in stage 10000 which took 3 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Starting lifecycle stage 10000 took 3.1189 Milliseconds
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Watchdog0 on managed thread 26
info: Orleans.Runtime.Messaging.IncomingMessageAcceptor[0]
Starting AsyncAgent Runtime.Messaging.GatewayAcceptor on managed thread 24
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Messaging.GatewayAcceptor0 on managed thread 27
info: Runtime.Messaging.GatewayClientCleanupAgent[0]
Starting AsyncAgent Runtime.Messaging.GatewayClientCleanupAgent on managed thread 24
info: Orleans.Runtime.Silo[100452]
Start gateway took 5 Milliseconds to finish
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Messaging.GatewayClientCleanupAgent0 on managed thread 28
info: Orleans.Runtime.MembershipService.MembershipOracleData[100604]
-BecomeActive
info: Orleans.Runtime.MembershipService.MembershipOracleData[100614]
About to send pings to 0 nodes in order to validate communication in the Joining state. Pinged nodes = []
info: Orleans.Runtime.TypeManager[0]
Expediting cluster type map refresh due to new silo, S172.19.0.3:11111:291747614
info: Orleans.Runtime.MembershipService.MembershipOracleData[100634]
-ReadAll (called from BecomeActive) Membership table 4 silos, 1 are Active, 3 are Dead, Version=<0, 0>. All silos: [SiloAddress=S172.19.0.3:11111:291747614 SiloName=Silo_787b0 Status=Active, SiloAddress=S172.19.0.3:11111:291745222 SiloName=Silo_39b85 Status=Dead, SiloAddress=S172.19.0.3:11111:291745251 SiloName=Silo_4aeb2 Status=Dead, SiloAddress=S172.19.0.3:11111:291747572 SiloName=Silo_efcb6 Status=Dead]
info: Orleans.Runtime.MembershipService.MembershipOracleData[100605]
-Finished BecomeActive.
info: Orleans.Runtime.Silo[100452]
Starting local silo status oracle took 46 Milliseconds to finish
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.Silo started in stage 19999 which took 59 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Starting lifecycle stage 19999 took 59.0451 Milliseconds
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.Silo started in stage 20000 which took 6 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Starting lifecycle stage 20000 took 6.0892 Milliseconds
Press ENTER to exit
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.Silo stopped in stage 20000 which took 11 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Stopping lifecycle stage 20000 took 18.4135 Milliseconds
info: Orleans.Runtime.Silo[100417]
Silo starting to Shutdown()
info: Orleans.Runtime.MembershipService.MembershipOracleData[100606]
-ShutDown
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.GrainDirectory.GlobalSingleInstanceActivationMaintainer0 on managed thread 22
info: Orleans.Runtime.Catalog[100545]
DeactivateAllActivations.
warn: Orleans.Runtime.Catalog[100506]
AllActiveSilos SiloStatusOracle.GetApproximateSiloStatuses empty
info: Orleans.Runtime.Dispatcher[100147]
Intermediate NonExistentActivation for message NewPlacement Request S172.19.0.3:11111:291747614*stg/17/00000011@S00000011->S172.19.0.3:11111:291747614*grn/13BAB4D8/00000000+OrleansCluster@a52b7585 #2: , with Exception Orleans.Runtime.Catalog+NonExistentActivationException: Non-existent activation: [ActivationAddress: S172.19.0.3:11111:291747614*grn/13BAB4D8/00000000+OrleansCluster@a52b7585, Full GrainId: [GrainId: *grn/13BAB4D8/000000000000000000000000000000000600000013bab4d8+OrleansCluster-0x344B5B0A, IdCategory: KeyExtGrain, BaseTypeCode: 331003096 (x13BAB4D8), PrimaryKey: 0 (x0), UniformHashCode: 877353738 (0x344B5B0A), KeyExtension: OrleansCluster], Full ActivationId: @4c461b65e0c7edc76aca7c5ca52b75850000000000000000], grain type: Orleans.Runtime.Versions.VersionStoreGrain.
at Orleans.Runtime.Catalog.GetOrCreateActivation(ActivationAddress address, Boolean newPlacement, String grainType, String genericArguments, Dictionary`2 requestContextData, Task& activatedPromise)
at Orleans.Runtime.Dispatcher.ReceiveMessage(Message message)
warn: Orleans.Runtime.Dispatcher[101535]
Failed to un-register NonExistentActivation S172.19.0.3:11111:291747614*grn/13BAB4D8/00000000+OrleansCluster@a52b7585
System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.UnregisterAsync(ActivationAddress address, UnregistrationCause cause, Int32 hopCount)
at Orleans.Runtime.Dispatcher.<>c__DisplayClass18_1.<<ReceiveMessage>b__0>d.MoveNext()
info: Orleans.Runtime.Dispatcher[101031]
Trying to forward after Non-existent activation, ForwardCount = 0. OldAddress = S172.19.0.3:11111:291747614*grn/13BAB4D8/00000000+OrleansCluster@a52b7585, ForwardingAddress = (null), Message NewPlacement Request S172.19.0.3:11111:291747614*stg/17/00000011@S00000011->S172.19.0.3:11111:291747614*grn/13BAB4D8/00000000+OrleansCluster@a52b7585 #2: , Exception: (null).
fail: Orleans.Runtime.Dispatcher[100071]
SelectTarget failed with Grain directory is stopping
System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)
at Orleans.Runtime.Placement.RandomPlacementDirector.OnSelectActivation(PlacementStrategy strategy, GrainId target, IPlacementRuntime context)
at Orleans.Runtime.Placement.PlacementDirectorsManager.SelectOrAddActivation(ActivationAddress sendingAddress, PlacementTarget targetGrain, IPlacementRuntime context, PlacementStrategy strategy)
at Orleans.Runtime.Dispatcher.AddressMessageAsync(Message message, PlacementTarget target, PlacementStrategy strategy, ActivationAddress targetAddress)
at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<<AsyncSendMessage>b__1>d.MoveNext()
fail: Orleans.Runtime.Dispatcher[100071]
SelectTarget failed with Grain directory is stopping
System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.Placement.RandomPlacementDirector.OnSelectActivation(PlacementStrategy strategy, GrainId target, IPlacementRuntime context)
at Orleans.Runtime.Placement.PlacementDirectorsManager.SelectOrAddActivation(ActivationAddress sendingAddress, PlacementTarget targetGrain, IPlacementRuntime context, PlacementStrategy strategy)
at Orleans.Runtime.Dispatcher.AddressMessageAsync(Message message, PlacementTarget target, PlacementStrategy strategy, ActivationAddress targetAddress)
at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<<AsyncSendMessage>b__1>d.MoveNext()
fail: Orleans.Runtime.Dispatcher[100071]
SelectTarget failed with Grain directory is stopping
System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.Placement.RandomPlacementDirector.OnSelectActivation(PlacementStrategy strategy, GrainId target, IPlacementRuntime context)
at Orleans.Runtime.Placement.PlacementDirectorsManager.SelectOrAddActivation(ActivationAddress sendingAddress, PlacementTarget targetGrain, IPlacementRuntime context, PlacementStrategy strategy)
at Orleans.Runtime.Dispatcher.AddressMessageAsync(Message message, PlacementTarget target, PlacementStrategy strategy, ActivationAddress targetAddress)
at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<<AsyncSendMessage>b__1>d.MoveNext()
fail: Orleans.Runtime.Dispatcher[100071]
SelectTarget failed with Grain directory is stopping
System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.Placement.RandomPlacementDirector.OnSelectActivation(PlacementStrategy strategy, GrainId target, IPlacementRuntime context)
at Orleans.Runtime.Placement.PlacementDirectorsManager.SelectOrAddActivation(ActivationAddress sendingAddress, PlacementTarget targetGrain, IPlacementRuntime context, PlacementStrategy strategy)
at Orleans.Runtime.Dispatcher.AddressMessageAsync(Message message, PlacementTarget target, PlacementStrategy strategy, ActivationAddress targetAddress)
at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<<AsyncSendMessage>b__1>d.MoveNext()
fail: Orleans.Runtime.Dispatcher[100071]
SelectTarget failed with Grain directory is stopping
System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.Placement.RandomPlacementDirector.OnSelectActivation(PlacementStrategy strategy, GrainId target, IPlacementRuntime context)
at Orleans.Runtime.Placement.PlacementDirectorsManager.SelectOrAddActivation(ActivationAddress sendingAddress, PlacementTarget targetGrain, IPlacementRuntime context, PlacementStrategy strategy)
at Orleans.Runtime.Dispatcher.AddressMessageAsync(Message message, PlacementTarget target, PlacementStrategy strategy, ActivationAddress targetAddress)
at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<<AsyncSendMessage>b__1>d.MoveNext()
fail: Orleans.Runtime.Dispatcher[100071]
SelectTarget failed with Grain directory is stopping
System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.Placement.RandomPlacementDirector.OnSelectActivation(PlacementStrategy strategy, GrainId target, IPlacementRuntime context)
at Orleans.Runtime.Placement.PlacementDirectorsManager.SelectOrAddActivation(ActivationAddress sendingAddress, PlacementTarget targetGrain, IPlacementRuntime context, PlacementStrategy strategy)
at Orleans.Runtime.Dispatcher.AddressMessageAsync(Message message, PlacementTarget target, PlacementStrategy strategy, ActivationAddress targetAddress)
at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<<AsyncSendMessage>b__1>d.MoveNext()
fail: Orleans.Runtime.Dispatcher[100071]
SelectTarget failed with Grain directory is stopping
System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.Placement.RandomPlacementDirector.OnSelectActivation(PlacementStrategy strategy, GrainId target, IPlacementRuntime context)
at Orleans.Runtime.Placement.PlacementDirectorsManager.SelectOrAddActivation(ActivationAddress sendingAddress, PlacementTarget targetGrain, IPlacementRuntime context, PlacementStrategy strategy)
at Orleans.Runtime.Dispatcher.AddressMessageAsync(Message message, PlacementTarget target, PlacementStrategy strategy, ActivationAddress targetAddress)
at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<<AsyncSendMessage>b__1>d.MoveNext()
fail: Orleans.Runtime.Dispatcher[100071]
SelectTarget failed with Grain directory is stopping
System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.Placement.RandomPlacementDirector.OnSelectActivation(PlacementStrategy strategy, GrainId target, IPlacementRuntime context)
at Orleans.Runtime.Placement.PlacementDirectorsManager.SelectOrAddActivation(ActivationAddress sendingAddress, PlacementTarget targetGrain, IPlacementRuntime context, PlacementStrategy strategy)
at Orleans.Runtime.Dispatcher.AddressMessageAsync(Message message, PlacementTarget target, PlacementStrategy strategy, ActivationAddress targetAddress)
at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<<AsyncSendMessage>b__1>d.MoveNext()
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Messaging.GatewayAcceptor0 on managed thread 27
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.Silo stopped in stage 19999 which took 2081 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Stopping lifecycle stage 19999 took 2081.8125 Milliseconds
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.Versions.GrainVersionStore stopped in stage 10000 which took 0 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Stopping lifecycle stage 10000 took 0.2651 Milliseconds
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer HostedClient stopped in stage 8000 which took 6 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.Silo stopped in stage 8000 which took 0 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Stopping lifecycle stage 8000 took 6.4309 Milliseconds
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.Silo stopped in stage 4000 which took 3 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Stopping lifecycle stage 4000 took 3.12 Milliseconds
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.InsideRuntimeClient stopped in stage 2000 which took 1 Milliseconds.
info: Orleans.Runtime.Silo[100413]
Silo is Stopped()
info: Orleans.Runtime.MembershipService.MembershipOracleData[100609]
-KillMyself
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Messaging.IncomingMessageAgent/Ping0 on managed thread 17
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Messaging.IncomingMessageAgent/System0 on managed thread 18
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Messaging.IncomingMessageAgent/Application0 on managed thread 19
info: Orleans.Runtime.ActivationDirectory[100528]
ActivationDirectory.PrintActivationDirectory(): Size = 0, Directory:
[]
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Messaging.IncomingMessageAcceptor0 on managed thread 14
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Messaging.SiloMessageSender/SystemSender0 on managed thread 16
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Messaging.SiloMessageSender/PingSender0 on managed thread 15
info: Orleans.Runtime.CounterStatistic[100703]
Stopping Windows perf counter stats collection
fail: Orleans.Runtime.ClientObserverRegistrar[101317]
Directory.RegisterAsync S172.19.0.3:11111:291747614*cli/00d2a68b@f9370431 failed.
System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
at Orleans.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
fail: Orleans.Runtime.ClientObserverRegistrar[101319]
OnClientRefreshTimer has thrown an exceptions.
System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
at Orleans.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
at Orleans.Runtime.ClientObserverRegistrar.OnClientRefreshTimer(Object data)
fail: Orleans.Runtime.Dispatcher[100071]
SelectTarget failed with Grain directory is stopping
System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.Placement.RandomPlacementDirector.OnSelectActivation(PlacementStrategy strategy, GrainId target, IPlacementRuntime context)
at Orleans.Runtime.Placement.PlacementDirectorsManager.SelectOrAddActivation(ActivationAddress sendingAddress, PlacementTarget targetGrain, IPlacementRuntime context, PlacementStrategy strategy)
at Orleans.Runtime.Dispatcher.AddressMessageAsync(Message message, PlacementTarget target, PlacementStrategy strategy, ActivationAddress targetAddress)
at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<<AsyncSendMessage>b__1>d.MoveNext()
info: Orleans.Runtime.Messaging.OutboundMessageQueue[100112]
Message was queued for sending after outbound queue was stopped: Unrecoverable Rejection (info: System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.Placement.RandomPlacementDirector.OnSelectActivation(PlacementStrategy strategy, GrainId target, IPlacementRuntime context)
at Orleans.Runtime.Placement.PlacementDirectorsManager.SelectOrAddActivation(ActivationAddress sendingAddress, PlacementTarget targetGrain, IPlacementRuntime context, PlacementStrategy strategy)
at Orleans.Runtime.Dispatcher.AddressMessageAsync(Message message, PlacementTarget target, PlacementStrategy strategy, ActivationAddress targetAddress)
at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<<AsyncSendMessage>b__1>d.MoveNext()) Response S172.19.0.3:11111:291747614*grn/13BAB4D8/00000000+OrleansCluster->S172.19.0.3:11111:291747614*stg/17/00000011@S00000011 #10:
info: Orleans.RuntimeSiloLogStatistics[100704]
Statistics: ^^^
App.Requests.Latency.Average.Millis=22.000
App.Requests.LatencyHistogram.Millis=[0.8192:1.6383]=1, [1.6384:3.2767]=3, [3.2768:6.5535]=3, [104.8576:209.7151]=1,
App.Requests.TimedOut.Current=0
App.Requests.Total.Requests.Current=8
Catalog.Activation.Collection.NumberOfCollections.Current=1
Catalog.Activation.Created.Current=0
Catalog.Activation.CurrentCount=0
Catalog.Activation.Destroyed.Current=0
Catalog.Activation.FailedToActivate.Current=0
Catalog.Activation.NonExistentActivations.Current=1
ConsistentRing.AverageRangePercentage=100.000
ConsistentRing.MyRange.RingDistance=x100000000
ConsistentRing.MyRange.RingPercentage=100.000
ConsistentRing.Ring=[S172.19.0.3:11111:291747614 -> <MultiRange: Size=x100000000, %Ring=100.000%>]
ConsistentRing.RingSize=1
Directory.CacheSize=0
Directory.Lookups.Cache.HitRatio=0, Delta=0
Directory.Lookups.Cache.Issued.Current=0
Directory.Lookups.Cache.Successes.Current=0
Directory.Lookups.Full.Issued.Current=9
Directory.Lookups.Local.Issued.Current=9
Directory.Lookups.Local.Successes.Current=0
Directory.Lookups.LocalDirectory.Issued.Current=2
Directory.Lookups.LocalDirectory.Successes.Current=0
Directory.Lookups.Remote.Received.Current=0
Directory.Lookups.Remote.Sent.Current=0
Directory.PartitionSize=0
Directory.Registrations.Issued.Current=2
Directory.Registrations.Local.Current=1
Directory.Registrations.Remote.Received.Current=0
Directory.Registrations.Remote.Sent.Current=0
Directory.Registrations.SingleAct.Issued.Current=0
Directory.Registrations.SingleAct.Local.Current=0
Directory.Registrations.SingleAct.Remote.Received.Current=0
Directory.Registrations.SingleAct.Remote.Sent.Current=0
Directory.Ring=[S172.19.0.3:11111:291747614/762F81A6]
Directory.Ring.MyPortion.AverageRingPercentage=100.000
Directory.Ring.MyPortion.RingDistance=0
Directory.Ring.MyPortion.RingPercentage=0.000
Directory.Ring.MyPredecessors=[]
Directory.Ring.MySuccessors=[]
Directory.Ring.RingSize=1
Directory.UnRegistrations.Issued.Current=1
Directory.UnRegistrations.Local.Current=0
Directory.UnRegistrations.Remote.Received.Current=0
Directory.UnRegistrations.Remote.Sent.Current=0
Directory.UnRegistrationsMany.Issued.Current=0
Directory.UnRegistrationsMany.Remote.Received.Current=0
Directory.UnRegistrationsMany.Remote.Sent.Current=0
Directory.Validations.Cache.Received.Current=0
Directory.Validations.Cache.Sent.Current=0
Dispatcher.NewPlacement.Current=1
Gateway.ConnectedClients.Current=0
Gateway.LoadShedding.Current=0
Gateway.Received.Current=0
Membership.ActiveCluster=[]
Membership.ActiveClusterSize=0
MessageCenter.ReceiveQueueLength=0
MessageCenter.SendQueueLength=0
Messaging.Acceptor.CheckedInSocketEventArgs.Current=0
Messaging.Acceptor.CheckedOutSocketEventArgs.Current=0
Messaging.Acceptor.InUseSocketEventArgs=0
Messaging.Expired.AtDispatch.Current=0
Messaging.Expired.AtInvoke.Current=0
Messaging.Expired.AtReceive.Current=0
Messaging.Expired.AtRespond.Current=0
Messaging.Expired.AtSend.Current=0
Messaging.Processing.ActivationData.All=0
Messaging.Processing.Dispatcher.Processed.Errors.Direction.OneWay.Current=0
Messaging.Processing.Dispatcher.Processed.Errors.Direction.Request.Current=10
Messaging.Processing.Dispatcher.Processed.Errors.Direction.Response.Current=0
Messaging.Processing.Dispatcher.Processed.Ok.Direction.OneWay.Current=0
Messaging.Processing.Dispatcher.Processed.Ok.Direction.Request.Current=0
Messaging.Processing.Dispatcher.Processed.Ok.Direction.Response.Current=0
Messaging.Processing.Dispatcher.Processed.ReRoute.Direction.OneWay.Current=0
Messaging.Processing.Dispatcher.Processed.ReRoute.Direction.Request.Current=1
Messaging.Processing.Dispatcher.Processed.ReRoute.Direction.Response.Current=0
Messaging.Processing.Dispatcher.Processed.Total.Current=11
Messaging.Processing.Dispatcher.Received.Direction.OneWay.Current=0
Messaging.Processing.Dispatcher.Received.Direction.Request.Current=1
Messaging.Processing.Dispatcher.Received.Direction.Response.Current=0
Messaging.Processing.Dispatcher.Received.OnActivationContext.Current=0
Messaging.Processing.Dispatcher.Received.OnNullContext.Current=1
Messaging.Processing.Dispatcher.Received.Total.Current=1
Messaging.Processing.IGC.Forwarded.Current=1
Messaging.Processing.IGC.ReRoute.Current=0
Messaging.Processing.IGC.Resent.Current=0
Messaging.Processing.IMA.Enqueued.ToActivationContex.Current=0
Messaging.Processing.IMA.Enqueued.ToNullContex.Current=1
Messaging.Processing.IMA.Enqueued.ToSystemTargetContex.Current=8
Messaging.Processing.IMA.Received.Current=9
Messaging.Received.BatchSize.PerSocketDirection.GatewayToClient=0.000
Messaging.Received.BatchSize.PerSocketDirection.SiloToSilo=0.000
Messaging.Received.BatchSizeBytesHistogram.Bytes.PerSocketDirection.GatewayToClient=
Messaging.Received.BatchSizeBytesHistogram.Bytes.PerSocketDirection.SiloToSilo=
Messaging.Received.Bytes.Header.Current=0
Messaging.Received.Bytes.Total.Current=0
Messaging.Received.Direction.OneWay.Current=0
Messaging.Received.Direction.Request.Current=0
Messaging.Received.Direction.Response.Current=0
Messaging.Received.Messages.Total.Current=0
Messaging.Received.MessageSizeHistogram.Bytes=
Messaging.Rejected.Request.Current=9
Messaging.Rerouted.OneWay.Current=0
Messaging.Rerouted.Request.Current=0
Messaging.Rerouted.Response.Current=0
Messaging.Sent.BatchSize.PerSocketDirection.GatewayToClient=0.000
Messaging.Sent.BatchSize.PerSocketDirection.SiloToSilo=0.000
Messaging.Sent.BatchSizeBytesHistogram.Bytes.PerSocketDirection.GatewayToClient=
Messaging.Sent.BatchSizeBytesHistogram.Bytes.PerSocketDirection.SiloToSilo=
Messaging.Sent.Bytes.Header.Current=0
Messaging.Sent.Bytes.Total.Current=0
Messaging.Sent.Direction.OneWay.Current=0
Messaging.Sent.Direction.Request.Current=0
Messaging.Sent.Direction.Response.Current=0
Messaging.Sent.LocalMessages.Current=9
Messaging.Sent.Messages.Total.Current=0
Messaging.Sent.MessageSizeHistogram.Bytes=
Networking.Sockets.GatewayToClient.Duplex.Closed.Current=0
Networking.Sockets.GatewayToClient.Duplex.Opened.Current=0
Networking.Sockets.Silo.Receiving.Closed.Current=0
Networking.Sockets.Silo.Receiving.Opened.Current=0
Networking.Sockets.Silo.Sending.Closed.Current=0
Networking.Sockets.Silo.Sending.Opened.Current=0
Queues.QueueSize.Instantaneous.Scheduler.LevelOne=0
Runtime.IsOverloaded=False
Runtime.Threads.AsynchAgent.Runtime.GrainDirectory.AdaptiveDirectoryCacheMaintainer`10.Current=1
Runtime.Threads.AsynchAgent.Runtime.GrainDirectory.GlobalSingleInstanceActivationMaintainer0.Current=0
Runtime.Threads.AsynchAgent.Runtime.Messaging.GatewayAcceptor0.Current=0
Runtime.Threads.AsynchAgent.Runtime.Messaging.GatewayClientCleanupAgent0.Current=1
Runtime.Threads.AsynchAgent.Runtime.Messaging.IncomingMessageAcceptor0.Current=0
Runtime.Threads.AsynchAgent.Runtime.Messaging.IncomingMessageAgent/Application0.Current=0
Runtime.Threads.AsynchAgent.Runtime.Messaging.IncomingMessageAgent/Ping0.Current=0
Runtime.Threads.AsynchAgent.Runtime.Messaging.IncomingMessageAgent/System0.Current=0
Runtime.Threads.AsynchAgent.Runtime.Messaging.SiloMessageSender/PingSender0.Current=0
Runtime.Threads.AsynchAgent.Runtime.Messaging.SiloMessageSender/SystemSender0.Current=0
Runtime.Threads.AsynchAgent.Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue0.Current=1
Runtime.Threads.AsynchAgent.Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue1.Current=1
Runtime.Threads.AsynchAgent.Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue2.Current=1
Runtime.Threads.AsynchAgent.Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue3.Current=1
Runtime.Threads.AsynchAgent.Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.SystemQueue0.Current=1
Runtime.Threads.AsynchAgent.Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.SystemQueue1.Current=1
Runtime.Threads.AsynchAgent.Runtime.Watchdog0.Current=1
Runtime.Threads.AsynchAgent.TotalThreadsCreated.Current=17
Scheduler.NumLongQueueWaitTimes.Current=0
Scheduler.NumLongRunningTurns.Current=0
Scheduler.WorkItemGroupCount=12
Serialization.BufferPool.AllocatedBuffers.Current=250
Serialization.BufferPool.BuffersInPool=250
Serialization.BufferPool.CheckedInBuffers.Current=250
Serialization.BufferPool.CheckedOutBuffers.Current=250
Serialization.BufferPool.DroppedBuffers.Current=0
Serialization.BufferPool.DroppedTooLargeBuffers.Current=0
Serialization.BufferPool.InUse.AllocatedAndNotInPool_Buffers=0
Serialization.BufferPool.InUse.CheckedOutAndNotCheckedIn_Buffers=0
Silo.StartTime=2019-03-31 17:00:14.554 GMT
Storage.Activate.Errors.Current=0
Storage.Activate.Total.Current=0
Storage.Clear.Errors.Current=0
Storage.Clear.Latency=0 Secs
Storage.Clear.Total.Current=0
Storage.Read.Errors.Current=0
Storage.Read.Latency=0 Secs
Storage.Read.Total.Current=0
Storage.Write.Errors.Current=0
Storage.Write.Latency=0 Secs
Storage.Write.Total.Current=0
SystemTarget..Current=2
Watchdog.NumHealthChecks.Current=0
fail: Orleans.Runtime.HostedClient[100326]
RunClientMessagePump has thrown exception
System.OperationCanceledException: The operation was canceled.
at System.Collections.Concurrent.BlockingCollection`1.TryTakeWithNoTimeValidation(T& item, Int32 millisecondsTimeout, CancellationToken cancellationToken, CancellationTokenSource combinedTokenSource)
at System.Collections.Concurrent.BlockingCollection`1.TryTake(T& item, Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Collections.Concurrent.BlockingCollection`1.Take(CancellationToken cancellationToken)
at Orleans.Runtime.HostedClient.RunClientMessagePump()
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.Silo stopped in stage 2000 which took 112 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Stopping lifecycle stage 2000 took 112.9731 Milliseconds
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.SiloOptionsLogger stopped in stage -2147483648 which took 0 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Stopping lifecycle stage -2147483648 took 0.2003 Milliseconds
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue1 on managed thread 11
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue1 on managed thread 11
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue1 on managed thread 11
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.SystemQueue1 on managed thread 9
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue1 on managed thread 11
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.SystemQueue1 on managed thread 9
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue0 on managed thread 10
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue3 on managed thread 13
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue3 on managed thread 13
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.SystemQueue0 on managed thread 8
info: Orleans.Runtime.Scheduler.OrleansTaskScheduler[101203]
OrleansTaskScheduler.PrintStatistics(): RunQueue=0, WorkItems=12, Directory:
[--System*WorkItemGroup:Name=*stg/10/0000000a,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=0; Total processed=0; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-5:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.19.0.3:11111:291747614*stg/10/0000000a@S0000000a] CurrentlyExecuting=null>
--System*WorkItemGroup:Name=*stg/11/0000000b,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=1; Total processed=1; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-6:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.19.0.3:11111:291747614*stg/11/0000000b@S0000000b] CurrentlyExecuting=null>
--System*WorkItemGroup:Name=*stg/12/0000000c,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=0; Total processed=0; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-2:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.19.0.3:11111:291747614*stg/12/0000000c@S0000000c] CurrentlyExecuting=null>
--System*WorkItemGroup:Name=*stg/13/0000000d,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=5; Total processed=5; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-8:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.19.0.3:11111:291747614*stg/13/0000000d@S0000000d] CurrentlyExecuting=null>
--System*WorkItemGroup:Name=*stg/14/0000000e,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=13; Total processed=13; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-11:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.19.0.3:11111:291747614*stg/14/0000000e@S0000000e] CurrentlyExecuting=null>
--System*WorkItemGroup:Name=*stg/15/0000000f,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=19; Total processed=19; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-10:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.19.0.3:11111:291747614*stg/15/0000000f@S0000000f] CurrentlyExecuting=null>
--System*WorkItemGroup:Name=*stg/17/00000011,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=29; Total processed=29; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-9:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.19.0.3:11111:291747614*stg/17/00000011@S00000011] CurrentlyExecuting=Unrecoverable Rejection (info: System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.Placement.RandomPlacementDirector.OnSelectActivation(PlacementStrategy strategy, GrainId target, IPlacementRuntime context)
at Orleans.Runtime.Placement.PlacementDirectorsManager.SelectOrAddActivation(ActivationAddress sendingAddress, PlacementTarget targetGrain, IPlacementRuntime context, PlacementStrategy strategy)
at Orleans.Runtime.Dispatcher.AddressMessageAsync(Message message, PlacementTarget target, PlacementStrategy strategy, ActivationAddress targetAddress)
at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<<AsyncSendMessage>b__1>d.MoveNext()) Response S172.19.0.3:11111:291747614*grn/13BAB4D8/00000000+OrleansCluster->S172.19.0.3:11111:291747614*stg/17/00000011@S00000011 #9: >
--System*WorkItemGroup:Name=*stg/19/00000013,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=0; Total processed=0; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-12:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.19.0.3:11111:291747614*stg/19/00000013@S00000013] CurrentlyExecuting=null>
--System*WorkItemGroup:Name=*stg/20/00000014,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=11; Total processed=11; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-1:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.19.0.3:11111:291747614*stg/20/00000014@S00000014] CurrentlyExecuting=null>
--System*WorkItemGroup:Name=*stg/22/00000016,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=5; Total processed=5; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-4:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.19.0.3:11111:291747614*stg/22/00000016@S00000016] CurrentlyExecuting=null>
--System*WorkItemGroup:Name=*stg/24/00000018,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=0; Total processed=0; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-7:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.19.0.3:11111:291747614*stg/24/00000018@S00000018] CurrentlyExecuting=null>
--System*WorkItemGroup:Name=*stg/27/0000001b,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=0; Total processed=0; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-3:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.19.0.3:11111:291747614*stg/27/0000001b@S0000001b] CurrentlyExecuting=null>]
warn: Orleans.Runtime.Silo[100220]
Process is exiting
info: Orleans.Runtime.Silo[100427]
Silo termination is in progress - Will wait for it to finish
warn: Orleans.Runtime.NoOpHostEnvironmentStatistics[100708]
No implementation of IHostEnvironmentStatistics was found. Load shedding will not work yet
info: Orleans.Runtime.Scheduler.OrleansTaskScheduler[0]
Starting OrleansTaskScheduler with 4 Max Active application Threads and 2 system thread.
info: Orleans.Runtime.Silo[100404]
Silo starting with GC settings: ServerGC=False GCLatencyMode=Interactive
warn: Orleans.Runtime.Silo[100405]
Note: Silo not running with ServerGC turned on - recommend checking app config : <configuration>-<runtime>-<gcServer enabled="true">
warn: Orleans.Runtime.Silo[100405]
Note: ServerGC only kicks in on multi-core systems (settings enabling ServerGC have no effect on single-core machines).
info: Orleans.Runtime.Silo[100403]
-------------- Initializing silo on host 0091df5ace4f MachineName 0091df5ace4f at 172.19.0.3:11111, gen 291748221 --------------
info: Orleans.Runtime.Silo[100415]
Starting silo Silo_e059b
info: Orleans.Runtime.Messaging.IncomingMessageAcceptor[101010]
Opened a listening socket at address 0.0.0.0:11111
info: Orleans.Runtime.Messaging.IncomingMessageAcceptor[101010]
Opened a listening socket at address 0.0.0.0:30000
info: Orleans.Runtime.ConsistentRing.VirtualBucketsRingProvider[0]
Starting VirtualBucketsRingProvider on silo S172.19.0.3:11111:291748221/x9BD288E4.
info: Orleans.Runtime.ConsistentRing.VirtualBucketsRingProvider[103003]
Added Server S172.19.0.3:11111:291748221/x9BD288E4. Current view: [S172.19.0.3:11111:291748221 -> <MultiRange: Size=x100000000, %Ring=100.000%>]
info: Orleans.Runtime.ConsistentRing.VirtualBucketsRingProvider[103005]
-NotifyLocalRangeSubscribers about old <(0 0], Size=x100000000, %Ring=100.000%> new <MultiRange: Size=x100000000, %Ring=100.000%> increased? True
info: Orleans.Runtime.Silo[0]
Skip multicluster oracle creation (no multicluster network configured)
info: Orleans.Runtime.Silo[100422]
-------------- Started silo S172.19.0.3:11111:291748221, ConsistentHashCode 9BD288E4 --------------
info: Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.SystemQueue[0]
Starting AsyncAgent Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.SystemQueue on managed thread 1
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.SystemQueue0 on managed thread 7
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.SystemQueue1 on managed thread 8
info: Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue[0]
Starting AsyncAgent Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue on managed thread 1
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue0 on managed thread 10
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue1 on managed thread 11
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue2 on managed thread 12
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue3 on managed thread 13
info: Orleans.Runtime.Silo[100452]
Start Scheduler took 38 Milliseconds to finish
info: Orleans.Runtime.SiloLifecycleSubject[100453]
Stage -2147483648: Orleans.Runtime.SiloOptionsLogger
info: Orleans.Runtime.SiloLifecycleSubject[100453]
Stage 2000: Orleans.Runtime.InsideRuntimeClient, Orleans.Runtime.Silo
info: Orleans.Runtime.SiloLifecycleSubject[100453]
Stage 4000: Orleans.Runtime.Silo
info: Orleans.Runtime.SiloLifecycleSubject[100453]
Stage 8000: HostedClient, Orleans.Runtime.Silo
info: Orleans.Runtime.SiloLifecycleSubject[100453]
Stage 10000: Orleans.Runtime.Versions.GrainVersionStore
info: Orleans.Runtime.SiloLifecycleSubject[100453]
Stage 19999: Orleans.Runtime.Silo
info: Orleans.Runtime.SiloLifecycleSubject[100453]
Stage 20000: Orleans.Runtime.Silo
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.ActivationCountBasedPlacementOptions:
ChooseOutOf: 2
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.ClusterMembershipOptions:
NumMissedTableIAmAliveLimit: 2
LivenessEnabled: True
ProbeTimeout: 00:00:10
TableRefreshTimeout: 00:01:00
DeathVoteExpirationTimeout: 00:02:00
IAmAliveTablePublishTimeout: 00:05:00
MaxJoinAttemptTime: 00:05:00
ExpectedClusterSize: 20
ValidateInitialConnectivity: True
UseLivenessGossip: True
NumProbedSilos: 3
NumMissedProbesLimit: 3
NumVotesForDeathDeclaration: 2
DefunctSiloExpiration: 7.00:00:00
DefunctSiloCleanupPeriod:
IsRunningAsUnitTest: False
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.ConsistentRingOptions:
NumVirtualBucketsConsistentRing: 30
UseVirtualBucketsConsistentRing: True
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.EndpointOptions:
AdvertisedIPAddress: 172.19.0.3
SiloPort: 11111
GatewayPort: 30000
SiloListeningEndpoint: 0.0.0.0:11111
GatewayListeningEndpoint: 0.0.0.0:30000
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.GrainCollectionOptions:
CollectionQuantum: 00:01:00
CollectionAge: 02:00:00
DeactivationTimeout: 00:00:30
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.GrainDirectoryOptions:
CachingStrategy: Adaptive
CacheSize: 1000000
InitialCacheTTL: 00:00:30
MaximumCacheTTL: 00:04:00
CacheTTLExtensionFactor: 2
LazyDeregistrationDelay: 00:01:00
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.GrainVersioningOptions:
DefaultCompatibilityStrategy: BackwardCompatible
DefaultVersionSelectorStrategy: AllCompatibleVersions
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.LoadSheddingOptions:
LoadSheddingEnabled: False
LoadSheddingLimit: 95
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.MemoryGrainStorageOptions-Default:
NumStorageGrains: 10
InitStage: 10000
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.MultiClusterOptions:
HasMultiClusterNetwork: False
MaxMultiClusterGateways: 10
BackgroundGossipInterval: 00:00:30
UseGlobalSingleInstanceByDefault: True
GlobalSingleInstanceNumberRetries: 10
GlobalSingleInstanceRetryInterval: 00:00:30
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.NetworkingOptions:
OpenConnectionTimeout: 00:00:05
MaxSocketAge: 10675199.02:48:05.4775807
MaxSockets: 500
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.PerformanceTuningOptions:
DefaultConnectionLimit: 200
Expect100Continue: False
UseNagleAlgorithm: False
MinDotNetThreadPoolSize: 200
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.ProcessExitHandlingOptions:
FastKillOnProcessExit: True
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.SchedulingOptions:
PerformDeadlockDetection: False
AllowCallChainReentrancy: True
MaxActiveThreads: 4
DelayWarningThreshold: 00:00:10
ActivationSchedulingQuantum: 00:00:00.1000000
TurnWarningLengthThreshold: 00:00:00.2000000
MaxPendingWorkItemsSoftLimit: 0
EnableWorkerThreadInjection: False
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.SerializationProviderOptions:
FallbackSerializationProvider:
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.SiloMessagingOptions:
SiloSenderQueues: 2
GatewaySenderQueues: 2
MaxForwardCount: 2
ClientDropTimeout: 00:01:00
ClientRegistrationRefresh: 00:05:00
MaxEnqueuedRequestsSoftLimit: 0
MaxEnqueuedRequestsHardLimit: 0
MaxEnqueuedRequestsSoftLimit_StatelessWorker: 0
MaxEnqueuedRequestsHardLimit_StatelessWorker: 0
MaxRequestProcessingTime: 02:00:00
AssumeHomogenousSilosForTesting: False
DEFAULT_SHUTDOWN_REROUTE_TIMEOUT: 00:00:10
ShutdownRerouteTimeout: 00:00:10
ResponseTimeout: 00:00:30
ResponseTimeoutWithDebugger: 00:30:00
MaxResendCount: 0
ResendOnTimeout: False
DropExpiredMessages: True
BufferPoolBufferSize: 4096
BufferPoolMaxSize: 10000
BufferPoolPreallocationSize: 250
PropagateActivityId: False
LargeMessageWarningThreshold: 85000
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.SiloOptions:
SiloName: Silo_e059b
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.StatisticsOptions:
PerfCountersWriteInterval: 00:00:30
LogWriteInterval: 00:05:00
CollectionLevel: Info
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.TelemetryOptions:
info: Orleans.Runtime.SiloOptionsLogger[0]
Configuration Orleans.Configuration.TypeManagementOptions:
TypeMapRefreshInterval: 00:01:00
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.SiloOptionsLogger started in stage -2147483648 which took 26 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Starting lifecycle stage -2147483648 took 33.1723 Milliseconds
info: Orleans.Runtime.Silo[100401]
Silo Start()
info: Orleans.Runtime.Silo[100430]
Configured ThreadPool.SetMinThreads() to values: 200,200. Previous values are: 2,2.
info: Orleans.Runtime.Silo[100436]
Configured .NET ServicePointManager to Expect100Continue=False, DefaultConnectionLimit=200, UseNagleAlgorithm=False to improve Azure storage performance.
info: Orleans.Runtime.Silo[100452]
ConfigureThreadPoolAndServicePointSettings took 3 Milliseconds to finish
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.Silo started in stage 2000 which took 6 Milliseconds.
info: Orleans.Runtime.GrainTypeManager[101711]
Loaded grain type summary for 8 types:
Grain class cart.grain.cart.grain.CartGrain [-1141067427 (0xBBFCB15D)] from cart.grain.dll implementing interfaces: cart.grain.ICart [1651422567 (0x626EB567)]
Grain class Orleans.Providers.Orleans.Providers.MemoryStreamQueueGrain [1359359215 (0x51062CEF)] from OrleansProviders.dll implementing interfaces: Orleans.Providers.IMemoryStreamQueueGrain [1158152105 (0x4507FFA9)]
Grain class Orleans.Runtime.Development.Orleans.Runtime.Development.DevelopmentLeaseProviderGrain [940524075 (0x380F422B)] from Orleans.Runtime.dll implementing interfaces: Orleans.Runtime.Development.IDevelopmentLeaseProviderGrain [-1234611069 (0xB6695483)]
Grain class Orleans.Runtime.Management.Orleans.Runtime.Management.ManagementGrain [1954798034 (0x7483D9D2)] from Orleans.Runtime.dll implementing interfaces: Orleans.Runtime.IManagementGrain [-1734666656 (0x989B1660)]
Grain class Orleans.Runtime.ReminderService.Orleans.Runtime.ReminderService.GrainBasedReminderTable [-55315191 (0xFCB3F509)] from Orleans.Runtime.dll implementing interfaces: Orleans.IReminderTableGrain [-1135060418 (0xBC585A3E)]
Grain class Orleans.Runtime.Versions.Orleans.Runtime.Versions.VersionStoreGrain [331003096 (0x13BAB4D8)] from Orleans.Runtime.dll implementing interfaces: Orleans.Runtime.Versions.IVersionStoreGrain [-297600501 (0xEE42FA0B)], Orleans.IGrainWithStringKey [-1277021679 (0xB3E23211)]
Grain class Orleans.Storage.Orleans.Storage.MemoryStorageGrain [819348208 (0x30D642F0)] from OrleansProviders.dll implementing interfaces: Orleans.Storage.IMemoryStorageGrain [577125491 (0x22663C73)]
Grain class Orleans.Streams.Orleans.Streams.PubSubRendezvousGrain [1903070868 (0x716E8E94)] from Orleans.Runtime.dll implementing interfaces: Orleans.Streams.IPubSubRendezvousGrain [1746702088 (0x681C8F08)]
info: Orleans.Runtime.InsideRuntimeClient[100452]
Start InsideRuntimeClient took 14 Milliseconds
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.InsideRuntimeClient started in stage 2000 which took 17 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Starting lifecycle stage 2000 took 24.5147 Milliseconds
info: Orleans.Runtime.Messaging.IncomingMessageAcceptor[0]
Starting AsyncAgent Runtime.Messaging.IncomingMessageAcceptor on managed thread 9
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Messaging.IncomingMessageAcceptor0 on managed thread 14
info: Runtime.Messaging.SiloMessageSender/PingSender[0]
Starting AsyncAgent Runtime.Messaging.SiloMessageSender/PingSender on managed thread 9
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Messaging.SiloMessageSender/PingSender0 on managed thread 15
info: Runtime.Messaging.SiloMessageSender/SystemSender[0]
Starting AsyncAgent Runtime.Messaging.SiloMessageSender/SystemSender on managed thread 9
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Messaging.SiloMessageSender/SystemSender0 on managed thread 17
info: Orleans.Runtime.Silo[100452]
Start Message center took 22 Milliseconds to finish
info: Runtime.Messaging.IncomingMessageAgent/Ping[0]
Starting AsyncAgent Runtime.Messaging.IncomingMessageAgent/Ping on managed thread 9
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Messaging.IncomingMessageAgent/Ping0 on managed thread 18
info: Runtime.Messaging.IncomingMessageAgent/System[0]
Starting AsyncAgent Runtime.Messaging.IncomingMessageAgent/System on managed thread 9
info: Runtime.Messaging.IncomingMessageAgent/Application[0]
Starting AsyncAgent Runtime.Messaging.IncomingMessageAgent/Application on managed thread 9
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Messaging.IncomingMessageAgent/System0 on managed thread 19
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Messaging.IncomingMessageAgent/Application0 on managed thread 20
info: Orleans.Runtime.Silo[100452]
Start Incoming message agents took 12 Milliseconds to finish
info: Orleans.Runtime.GrainDirectory.LocalGrainDirectory[0]
Start
info: Runtime.GrainDirectory.AdaptiveDirectoryCacheMaintainer`1[0]
Starting AsyncAgent Runtime.GrainDirectory.AdaptiveDirectoryCacheMaintainer`1 on managed thread 9
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.GrainDirectory.AdaptiveDirectoryCacheMaintainer`10 on managed thread 21
info: Runtime.GrainDirectory.GlobalSingleInstanceActivationMaintainer[0]
Starting AsyncAgent Runtime.GrainDirectory.GlobalSingleInstanceActivationMaintainer on managed thread 9
info: Orleans.Runtime.Silo[100452]
Start local grain directory took 13 Milliseconds to finish
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.GrainDirectory.GlobalSingleInstanceActivationMaintainer0 on managed thread 22
info: Orleans.Runtime.Silo[100452]
Init implicit stream subscribe table took 13 Milliseconds to finish
info: Orleans.Runtime.Catalog[100507]
Before collection#1: memory=7MB, #activations=0, collector=<#Activations=0, #Buckets=0, buckets=[]>.
info: Orleans.Runtime.Silo[100452]
Create system targets and inject dependencies took 63 Milliseconds to finish
info: Orleans.Runtime.Catalog[100508]
After collection#1: memory=7MB, #activations=0, collected 0 activations, collector=<#Activations=0, #Buckets=0, buckets=[]>, collection time=00:00:00.0112799.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.Silo started in stage 4000 which took 138 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Starting lifecycle stage 4000 took 139.5226 Milliseconds
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer HostedClient started in stage 8000 which took 16 Milliseconds.
info: Orleans.Runtime.Silo[100452]
Init grain services took 2 Milliseconds to finish
info: Orleans.Runtime.MembershipService.MembershipOracleData[100603]
MembershipOracle starting on host = 0091df5ace4f address = S172.19.0.3:11111:291748221 at 2019-03-31 17:10:22.095 GMT, backOffMax = 00:00:20
info: Orleans.Runtime.MembershipService.MembershipOracleData[100652]
-CleanupTable called on silo startup. Membership table 4 silos, 0 are Active, 4 are Dead, Version=<0, 0>. All silos: [SiloAddress=S172.19.0.3:11111:291745222 SiloName=Silo_39b85 Status=Dead, SiloAddress=S172.19.0.3:11111:291745251 SiloName=Silo_4aeb2 Status=Dead, SiloAddress=S172.19.0.3:11111:291747572 SiloName=Silo_efcb6 Status=Dead, SiloAddress=S172.19.0.3:11111:291747614 SiloName=Silo_787b0 Status=Dead]
info: Orleans.Runtime.MembershipService.MembershipOracleData[100660]
Starting IAmAliveUpdateTimer.
info: Orleans.Runtime.Silo[100452]
Starting local silo status oracle took 5896 Milliseconds to finish
info: Orleans.Runtime.Silo[100452]
Init type manager took 13 Milliseconds to finish
info: Orleans.Runtime.CounterStatistic[100702]
Starting Windows perf counter stats collection with frequency=00:00:30
info: Orleans.Runtime.Silo[100452]
Start silo statistics took 3 Milliseconds to finish
info: Orleans.Runtime.DeploymentLoadPublisher[0]
Starting DeploymentLoadPublisher.
info: Orleans.Runtime.DeploymentLoadPublisher[0]
Started DeploymentLoadPublisher.
info: Orleans.Runtime.Silo[100452]
Start deployment load collector took 20 Milliseconds to finish
info: Orleans.Runtime.Watchdog[0]
Starting Silo Watchdog.
info: Runtime.Watchdog[0]
Starting AsyncAgent Runtime.Watchdog on managed thread 6
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.Silo started in stage 8000 which took 5945 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Starting lifecycle stage 8000 took 5977.0249 Milliseconds
info: Orleans.Storage.MemoryGrainStorage.Default[0]
Init: Name=Default NumStorageGrains=10
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.Versions.GrainVersionStore started in stage 10000 which took 2 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Starting lifecycle stage 10000 took 2.4858 Milliseconds
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Watchdog0 on managed thread 27
info: Orleans.Runtime.Messaging.IncomingMessageAcceptor[0]
Starting AsyncAgent Runtime.Messaging.GatewayAcceptor on managed thread 6
info: Runtime.Messaging.GatewayClientCleanupAgent[0]
Starting AsyncAgent Runtime.Messaging.GatewayClientCleanupAgent on managed thread 6
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Messaging.GatewayAcceptor0 on managed thread 28
info: Orleans.Runtime.Silo[100452]
Start gateway took 5 Milliseconds to finish
info: Orleans.Threading.ThreadPoolThread[0]
Starting thread Runtime.Messaging.GatewayClientCleanupAgent0 on managed thread 29
info: Orleans.Runtime.MembershipService.MembershipOracleData[100604]
-BecomeActive
info: Orleans.Runtime.MembershipService.MembershipOracleData[100614]
About to send pings to 0 nodes in order to validate communication in the Joining state. Pinged nodes = []
info: Orleans.Runtime.TypeManager[0]
Expediting cluster type map refresh due to new silo, S172.19.0.3:11111:291748221
info: Orleans.Runtime.MembershipService.MembershipOracleData[100634]
-ReadAll (called from BecomeActive) Membership table 5 silos, 1 are Active, 4 are Dead, Version=<0, 0>. All silos: [SiloAddress=S172.19.0.3:11111:291748221 SiloName=Silo_e059b Status=Active, SiloAddress=S172.19.0.3:11111:291745222 SiloName=Silo_39b85 Status=Dead, SiloAddress=S172.19.0.3:11111:291745251 SiloName=Silo_4aeb2 Status=Dead, SiloAddress=S172.19.0.3:11111:291747572 SiloName=Silo_efcb6 Status=Dead, SiloAddress=S172.19.0.3:11111:291747614 SiloName=Silo_787b0 Status=Dead]
info: Orleans.Runtime.MembershipService.MembershipOracleData[100605]
-Finished BecomeActive.
info: Orleans.Runtime.Silo[100452]
Starting local silo status oracle took 54 Milliseconds to finish
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.Silo started in stage 19999 which took 66 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Starting lifecycle stage 19999 took 67.2003 Milliseconds
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.Silo started in stage 20000 which took 7 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Starting lifecycle stage 20000 took 8.0363 Milliseconds
Press ENTER to exit
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.Silo stopped in stage 20000 which took 9 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Stopping lifecycle stage 20000 took 13.1184 Milliseconds
info: Orleans.Runtime.Silo[100417]
Silo starting to Shutdown()
info: Orleans.Runtime.MembershipService.MembershipOracleData[100606]
-ShutDown
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.GrainDirectory.GlobalSingleInstanceActivationMaintainer0 on managed thread 22
info: Orleans.Runtime.Catalog[100545]
DeactivateAllActivations.
fail: Orleans.Runtime.Dispatcher[100071]
SelectTarget failed with Grain directory is stopping
System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.Placement.RandomPlacementDirector.OnSelectActivation(PlacementStrategy strategy, GrainId target, IPlacementRuntime context)
at Orleans.Runtime.Placement.PlacementDirectorsManager.SelectOrAddActivation(ActivationAddress sendingAddress, PlacementTarget targetGrain, IPlacementRuntime context, PlacementStrategy strategy)
at Orleans.Runtime.Dispatcher.AddressMessageAsync(Message message, PlacementTarget target, PlacementStrategy strategy, ActivationAddress targetAddress)
at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<<AsyncSendMessage>b__1>d.MoveNext()
fail: Orleans.Runtime.Dispatcher[100071]
SelectTarget failed with Grain directory is stopping
System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.Placement.RandomPlacementDirector.OnSelectActivation(PlacementStrategy strategy, GrainId target, IPlacementRuntime context)
at Orleans.Runtime.Placement.PlacementDirectorsManager.SelectOrAddActivation(ActivationAddress sendingAddress, PlacementTarget targetGrain, IPlacementRuntime context, PlacementStrategy strategy)
at Orleans.Runtime.Dispatcher.AddressMessageAsync(Message message, PlacementTarget target, PlacementStrategy strategy, ActivationAddress targetAddress)
at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<<AsyncSendMessage>b__1>d.MoveNext()
fail: Orleans.Runtime.Dispatcher[100071]
SelectTarget failed with Grain directory is stopping
System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.Placement.RandomPlacementDirector.OnSelectActivation(PlacementStrategy strategy, GrainId target, IPlacementRuntime context)
at Orleans.Runtime.Placement.PlacementDirectorsManager.SelectOrAddActivation(ActivationAddress sendingAddress, PlacementTarget targetGrain, IPlacementRuntime context, PlacementStrategy strategy)
at Orleans.Runtime.Dispatcher.AddressMessageAsync(Message message, PlacementTarget target, PlacementStrategy strategy, ActivationAddress targetAddress)
at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<<AsyncSendMessage>b__1>d.MoveNext()
fail: Orleans.Runtime.Dispatcher[100071]
SelectTarget failed with Grain directory is stopping
System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.Placement.RandomPlacementDirector.OnSelectActivation(PlacementStrategy strategy, GrainId target, IPlacementRuntime context)
at Orleans.Runtime.Placement.PlacementDirectorsManager.SelectOrAddActivation(ActivationAddress sendingAddress, PlacementTarget targetGrain, IPlacementRuntime context, PlacementStrategy strategy)
at Orleans.Runtime.Dispatcher.AddressMessageAsync(Message message, PlacementTarget target, PlacementStrategy strategy, ActivationAddress targetAddress)
at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<<AsyncSendMessage>b__1>d.MoveNext()
fail: Orleans.Runtime.Dispatcher[100071]
SelectTarget failed with Grain directory is stopping
System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.Placement.RandomPlacementDirector.OnSelectActivation(PlacementStrategy strategy, GrainId target, IPlacementRuntime context)
at Orleans.Runtime.Placement.PlacementDirectorsManager.SelectOrAddActivation(ActivationAddress sendingAddress, PlacementTarget targetGrain, IPlacementRuntime context, PlacementStrategy strategy)
at Orleans.Runtime.Dispatcher.AddressMessageAsync(Message message, PlacementTarget target, PlacementStrategy strategy, ActivationAddress targetAddress)
at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<<AsyncSendMessage>b__1>d.MoveNext()
fail: Orleans.Runtime.Dispatcher[100071]
SelectTarget failed with Grain directory is stopping
System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.Placement.RandomPlacementDirector.OnSelectActivation(PlacementStrategy strategy, GrainId target, IPlacementRuntime context)
at Orleans.Runtime.Placement.PlacementDirectorsManager.SelectOrAddActivation(ActivationAddress sendingAddress, PlacementTarget targetGrain, IPlacementRuntime context, PlacementStrategy strategy)
at Orleans.Runtime.Dispatcher.AddressMessageAsync(Message message, PlacementTarget target, PlacementStrategy strategy, ActivationAddress targetAddress)
at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<<AsyncSendMessage>b__1>d.MoveNext()
fail: Orleans.Runtime.Dispatcher[100071]
SelectTarget failed with Grain directory is stopping
System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.Placement.RandomPlacementDirector.OnSelectActivation(PlacementStrategy strategy, GrainId target, IPlacementRuntime context)
at Orleans.Runtime.Placement.PlacementDirectorsManager.SelectOrAddActivation(ActivationAddress sendingAddress, PlacementTarget targetGrain, IPlacementRuntime context, PlacementStrategy strategy)
at Orleans.Runtime.Dispatcher.AddressMessageAsync(Message message, PlacementTarget target, PlacementStrategy strategy, ActivationAddress targetAddress)
at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<<AsyncSendMessage>b__1>d.MoveNext()
fail: Orleans.Runtime.Dispatcher[100071]
SelectTarget failed with Grain directory is stopping
System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.Placement.RandomPlacementDirector.OnSelectActivation(PlacementStrategy strategy, GrainId target, IPlacementRuntime context)
at Orleans.Runtime.Placement.PlacementDirectorsManager.SelectOrAddActivation(ActivationAddress sendingAddress, PlacementTarget targetGrain, IPlacementRuntime context, PlacementStrategy strategy)
at Orleans.Runtime.Dispatcher.AddressMessageAsync(Message message, PlacementTarget target, PlacementStrategy strategy, ActivationAddress targetAddress)
at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<<AsyncSendMessage>b__1>d.MoveNext()
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Messaging.GatewayAcceptor0 on managed thread 28
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.Silo stopped in stage 19999 which took 2055 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Stopping lifecycle stage 19999 took 2056.1719 Milliseconds
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.Versions.GrainVersionStore stopped in stage 10000 which took 0 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Stopping lifecycle stage 10000 took 0.4779 Milliseconds
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer HostedClient stopped in stage 8000 which took 6 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.Silo stopped in stage 8000 which took 0 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Stopping lifecycle stage 8000 took 6.5436 Milliseconds
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.Silo stopped in stage 4000 which took 2 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Stopping lifecycle stage 4000 took 2.2788 Milliseconds
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.InsideRuntimeClient stopped in stage 2000 which took 1 Milliseconds.
info: Orleans.Runtime.Silo[100413]
Silo is Stopped()
info: Orleans.Runtime.MembershipService.MembershipOracleData[100609]
-KillMyself
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Messaging.IncomingMessageAgent/Application0 on managed thread 20
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Messaging.IncomingMessageAgent/System0 on managed thread 19
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Messaging.IncomingMessageAgent/Ping0 on managed thread 18
info: Orleans.Runtime.ActivationDirectory[100528]
ActivationDirectory.PrintActivationDirectory(): Size = 0, Directory:
[]
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Messaging.IncomingMessageAcceptor0 on managed thread 14
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Messaging.SiloMessageSender/SystemSender0 on managed thread 17
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Messaging.SiloMessageSender/PingSender0 on managed thread 15
info: Orleans.Runtime.CounterStatistic[100703]
Stopping Windows perf counter stats collection
fail: Orleans.Runtime.ClientObserverRegistrar[101317]
Directory.RegisterAsync S172.19.0.3:11111:291748221*cli/7796229c@e2ed5b16 failed.
System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
at Orleans.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
fail: Orleans.Runtime.ClientObserverRegistrar[101319]
OnClientRefreshTimer has thrown an exceptions.
System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.RegisterAsync(ActivationAddress address, Boolean singleActivation, Int32 hopCount)
at Orleans.OrleansTaskExtentions.LogException(Task task, ILogger logger, ErrorCode errorCode, String message)
at Orleans.Runtime.ClientObserverRegistrar.OnClientRefreshTimer(Object data)
info: Orleans.RuntimeSiloLogStatistics[100704]
Statistics: ^^^
App.Requests.Latency.Average.Millis=16.000
App.Requests.LatencyHistogram.Millis=[0.8192:1.6383]=2, [1.6384:3.2767]=2, [3.2768:6.5535]=3, [104.8576:209.7151]=1,
App.Requests.TimedOut.Current=0
App.Requests.Total.Requests.Current=8
Catalog.Activation.Collection.NumberOfCollections.Current=1
Catalog.Activation.Created.Current=0
Catalog.Activation.CurrentCount=0
Catalog.Activation.Destroyed.Current=0
Catalog.Activation.FailedToActivate.Current=0
ConsistentRing.AverageRangePercentage=100.000
ConsistentRing.MyRange.RingDistance=x100000000
ConsistentRing.MyRange.RingPercentage=100.000
ConsistentRing.Ring=[S172.19.0.3:11111:291748221 -> <MultiRange: Size=x100000000, %Ring=100.000%>]
ConsistentRing.RingSize=1
Directory.CacheSize=0
Directory.Lookups.Cache.HitRatio=0, Delta=0
Directory.Lookups.Cache.Issued.Current=0
Directory.Lookups.Cache.Successes.Current=0
Directory.Lookups.Full.Issued.Current=8
Directory.Lookups.Local.Issued.Current=8
Directory.Lookups.Local.Successes.Current=0
Directory.Lookups.LocalDirectory.Issued.Current=1
Directory.Lookups.LocalDirectory.Successes.Current=0
Directory.Lookups.Remote.Received.Current=0
Directory.Lookups.Remote.Sent.Current=0
Directory.PartitionSize=0
Directory.Registrations.Issued.Current=2
Directory.Registrations.Local.Current=1
Directory.Registrations.Remote.Received.Current=0
Directory.Registrations.Remote.Sent.Current=0
Directory.Registrations.SingleAct.Issued.Current=0
Directory.Registrations.SingleAct.Local.Current=0
Directory.Registrations.SingleAct.Remote.Received.Current=0
Directory.Registrations.SingleAct.Remote.Sent.Current=0
Directory.Ring=[S172.19.0.3:11111:291748221/9BD288E4]
Directory.Ring.MyPortion.AverageRingPercentage=100.000
Directory.Ring.MyPortion.RingDistance=0
Directory.Ring.MyPortion.RingPercentage=0.000
Directory.Ring.MyPredecessors=[]
Directory.Ring.MySuccessors=[]
Directory.Ring.RingSize=1
Directory.UnRegistrations.Issued.Current=0
Directory.UnRegistrations.Local.Current=0
Directory.UnRegistrations.Remote.Received.Current=0
Directory.UnRegistrations.Remote.Sent.Current=0
Directory.UnRegistrationsMany.Issued.Current=0
Directory.UnRegistrationsMany.Remote.Received.Current=0
Directory.UnRegistrationsMany.Remote.Sent.Current=0
Directory.Validations.Cache.Received.Current=0
Directory.Validations.Cache.Sent.Current=0
Gateway.ConnectedClients.Current=0
Gateway.LoadShedding.Current=0
Gateway.Received.Current=0
Membership.ActiveCluster=[]
Membership.ActiveClusterSize=0
MessageCenter.ReceiveQueueLength=0
MessageCenter.SendQueueLength=0
Messaging.Acceptor.CheckedInSocketEventArgs.Current=0
Messaging.Acceptor.CheckedOutSocketEventArgs.Current=0
Messaging.Acceptor.InUseSocketEventArgs=0
Messaging.Expired.AtDispatch.Current=0
Messaging.Expired.AtInvoke.Current=0
Messaging.Expired.AtReceive.Current=0
Messaging.Expired.AtRespond.Current=0
Messaging.Expired.AtSend.Current=0
Messaging.Processing.ActivationData.All=0
Messaging.Processing.Dispatcher.Processed.Errors.Direction.OneWay.Current=0
Messaging.Processing.Dispatcher.Processed.Errors.Direction.Request.Current=8
Messaging.Processing.Dispatcher.Processed.Errors.Direction.Response.Current=0
Messaging.Processing.Dispatcher.Processed.Ok.Direction.OneWay.Current=0
Messaging.Processing.Dispatcher.Processed.Ok.Direction.Request.Current=0
Messaging.Processing.Dispatcher.Processed.Ok.Direction.Response.Current=0
Messaging.Processing.Dispatcher.Processed.ReRoute.Direction.OneWay.Current=0
Messaging.Processing.Dispatcher.Processed.ReRoute.Direction.Request.Current=0
Messaging.Processing.Dispatcher.Processed.ReRoute.Direction.Response.Current=0
Messaging.Processing.Dispatcher.Processed.Total.Current=8
Messaging.Processing.Dispatcher.Received.Direction.OneWay.Current=0
Messaging.Processing.Dispatcher.Received.Direction.Request.Current=0
Messaging.Processing.Dispatcher.Received.Direction.Response.Current=0
Messaging.Processing.Dispatcher.Received.OnActivationContext.Current=0
Messaging.Processing.Dispatcher.Received.OnNullContext.Current=0
Messaging.Processing.Dispatcher.Received.Total.Current=0
Messaging.Processing.IGC.Forwarded.Current=0
Messaging.Processing.IGC.ReRoute.Current=0
Messaging.Processing.IGC.Resent.Current=0
Messaging.Processing.IMA.Enqueued.ToActivationContex.Current=0
Messaging.Processing.IMA.Enqueued.ToNullContex.Current=0
Messaging.Processing.IMA.Enqueued.ToSystemTargetContex.Current=8
Messaging.Processing.IMA.Received.Current=8
Messaging.Received.BatchSize.PerSocketDirection.GatewayToClient=0.000
Messaging.Received.BatchSize.PerSocketDirection.SiloToSilo=0.000
Messaging.Received.BatchSizeBytesHistogram.Bytes.PerSocketDirection.GatewayToClient=
Messaging.Received.BatchSizeBytesHistogram.Bytes.PerSocketDirection.SiloToSilo=
Messaging.Received.Bytes.Header.Current=0
Messaging.Received.Bytes.Total.Current=0
Messaging.Received.Direction.OneWay.Current=0
Messaging.Received.Direction.Request.Current=0
Messaging.Received.Direction.Response.Current=0
Messaging.Received.Messages.Total.Current=0
Messaging.Received.MessageSizeHistogram.Bytes=
Messaging.Rejected.Request.Current=8
Messaging.Rerouted.OneWay.Current=0
Messaging.Rerouted.Request.Current=0
Messaging.Rerouted.Response.Current=0
Messaging.Sent.BatchSize.PerSocketDirection.GatewayToClient=0.000
Messaging.Sent.BatchSize.PerSocketDirection.SiloToSilo=0.000
Messaging.Sent.BatchSizeBytesHistogram.Bytes.PerSocketDirection.GatewayToClient=
Messaging.Sent.BatchSizeBytesHistogram.Bytes.PerSocketDirection.SiloToSilo=
Messaging.Sent.Bytes.Header.Current=0
Messaging.Sent.Bytes.Total.Current=0
Messaging.Sent.Direction.OneWay.Current=0
Messaging.Sent.Direction.Request.Current=0
Messaging.Sent.Direction.Response.Current=0
Messaging.Sent.LocalMessages.Current=8
Messaging.Sent.Messages.Total.Current=0
Messaging.Sent.MessageSizeHistogram.Bytes=
Networking.Sockets.GatewayToClient.Duplex.Closed.Current=0
Networking.Sockets.GatewayToClient.Duplex.Opened.Current=0
Networking.Sockets.Silo.Receiving.Closed.Current=0
Networking.Sockets.Silo.Receiving.Opened.Current=0
Networking.Sockets.Silo.Sending.Closed.Current=0
Networking.Sockets.Silo.Sending.Opened.Current=0
Queues.QueueSize.Instantaneous.Scheduler.LevelOne=0
Runtime.IsOverloaded=False
Runtime.Threads.AsynchAgent.Runtime.GrainDirectory.AdaptiveDirectoryCacheMaintainer`10.Current=1
Runtime.Threads.AsynchAgent.Runtime.GrainDirectory.GlobalSingleInstanceActivationMaintainer0.Current=0
Runtime.Threads.AsynchAgent.Runtime.Messaging.GatewayAcceptor0.Current=0
Runtime.Threads.AsynchAgent.Runtime.Messaging.GatewayClientCleanupAgent0.Current=1
Runtime.Threads.AsynchAgent.Runtime.Messaging.IncomingMessageAcceptor0.Current=0
Runtime.Threads.AsynchAgent.Runtime.Messaging.IncomingMessageAgent/Application0.Current=0
Runtime.Threads.AsynchAgent.Runtime.Messaging.IncomingMessageAgent/Ping0.Current=0
Runtime.Threads.AsynchAgent.Runtime.Messaging.IncomingMessageAgent/System0.Current=0
Runtime.Threads.AsynchAgent.Runtime.Messaging.SiloMessageSender/PingSender0.Current=0
Runtime.Threads.AsynchAgent.Runtime.Messaging.SiloMessageSender/SystemSender0.Current=0
Runtime.Threads.AsynchAgent.Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue0.Current=1
Runtime.Threads.AsynchAgent.Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue1.Current=1
Runtime.Threads.AsynchAgent.Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue2.Current=1
Runtime.Threads.AsynchAgent.Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue3.Current=1
Runtime.Threads.AsynchAgent.Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.SystemQueue0.Current=1
Runtime.Threads.AsynchAgent.Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.SystemQueue1.Current=1
Runtime.Threads.AsynchAgent.Runtime.Watchdog0.Current=1
Runtime.Threads.AsynchAgent.TotalThreadsCreated.Current=17
Scheduler.NumLongQueueWaitTimes.Current=0
Scheduler.NumLongRunningTurns.Current=0
Scheduler.WorkItemGroupCount=12
Serialization.BufferPool.AllocatedBuffers.Current=250
Serialization.BufferPool.BuffersInPool=250
Serialization.BufferPool.CheckedInBuffers.Current=250
Serialization.BufferPool.CheckedOutBuffers.Current=250
Serialization.BufferPool.DroppedBuffers.Current=0
Serialization.BufferPool.DroppedTooLargeBuffers.Current=0
Serialization.BufferPool.InUse.AllocatedAndNotInPool_Buffers=0
Serialization.BufferPool.InUse.CheckedOutAndNotCheckedIn_Buffers=0
Silo.StartTime=2019-03-31 17:10:22.109 GMT
Storage.Activate.Errors.Current=0
Storage.Activate.Total.Current=0
Storage.Clear.Errors.Current=0
Storage.Clear.Latency=0 Secs
Storage.Clear.Total.Current=0
Storage.Read.Errors.Current=0
Storage.Read.Latency=0 Secs
Storage.Read.Total.Current=0
Storage.Write.Errors.Current=0
Storage.Write.Latency=0 Secs
Storage.Write.Total.Current=0
SystemTarget..Current=2
Watchdog.NumHealthChecks.Current=0
fail: Orleans.Runtime.HostedClient[100326]
RunClientMessagePump has thrown exception
System.OperationCanceledException: The operation was canceled.
at System.Collections.Concurrent.BlockingCollection`1.TryTakeWithNoTimeValidation(T& item, Int32 millisecondsTimeout, CancellationToken cancellationToken, CancellationTokenSource combinedTokenSource)
at System.Collections.Concurrent.BlockingCollection`1.TryTake(T& item, Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Collections.Concurrent.BlockingCollection`1.Take(CancellationToken cancellationToken)
at Orleans.Runtime.HostedClient.RunClientMessagePump()
fail: Orleans.Runtime.Dispatcher[100071]
SelectTarget failed with Grain directory is stopping
System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.Placement.RandomPlacementDirector.OnSelectActivation(PlacementStrategy strategy, GrainId target, IPlacementRuntime context)
at Orleans.Runtime.Placement.PlacementDirectorsManager.SelectOrAddActivation(ActivationAddress sendingAddress, PlacementTarget targetGrain, IPlacementRuntime context, PlacementStrategy strategy)
at Orleans.Runtime.Dispatcher.AddressMessageAsync(Message message, PlacementTarget target, PlacementStrategy strategy, ActivationAddress targetAddress)
at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<<AsyncSendMessage>b__1>d.MoveNext()
info: Orleans.Runtime.Messaging.OutboundMessageQueue[100112]
Message was queued for sending after outbound queue was stopped: Unrecoverable Rejection (info: System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.Placement.RandomPlacementDirector.OnSelectActivation(PlacementStrategy strategy, GrainId target, IPlacementRuntime context)
at Orleans.Runtime.Placement.PlacementDirectorsManager.SelectOrAddActivation(ActivationAddress sendingAddress, PlacementTarget targetGrain, IPlacementRuntime context, PlacementStrategy strategy)
at Orleans.Runtime.Dispatcher.AddressMessageAsync(Message message, PlacementTarget target, PlacementStrategy strategy, ActivationAddress targetAddress)
at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<<AsyncSendMessage>b__1>d.MoveNext()) Response S172.19.0.3:11111:291748221*grn/13BAB4D8/00000000+OrleansCluster->S172.19.0.3:11111:291748221*stg/17/00000011@S00000011 #10:
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.Silo stopped in stage 2000 which took 85 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Stopping lifecycle stage 2000 took 85.4675 Milliseconds
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Lifecycle observer Orleans.Runtime.SiloOptionsLogger stopped in stage -2147483648 which took 0 Milliseconds.
info: Orleans.Runtime.SiloLifecycleSubject[100452]
Stopping lifecycle stage -2147483648 took 0.0132 Milliseconds
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue0 on managed thread 10
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue1 on managed thread 11
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue2 on managed thread 12
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.MainQueue3 on managed thread 13
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.SystemQueue1 on managed thread 8
info: Orleans.Threading.ThreadPoolThread[100328]
Stopping Thread Runtime.Scheduler.OrleansSchedulerAsynchAgent/Scheduler.LevelOne.SystemQueue0 on managed thread 7
info: Orleans.Runtime.Scheduler.OrleansTaskScheduler[101203]
OrleansTaskScheduler.PrintStatistics(): RunQueue=0, WorkItems=12, Directory:
[--System*WorkItemGroup:Name=*stg/10/0000000a,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=0; Total processed=0; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-5:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.19.0.3:11111:291748221*stg/10/0000000a@S0000000a] CurrentlyExecuting=null>
--System*WorkItemGroup:Name=*stg/11/0000000b,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=1; Total processed=1; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-6:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.19.0.3:11111:291748221*stg/11/0000000b@S0000000b] CurrentlyExecuting=null>
--System*WorkItemGroup:Name=*stg/12/0000000c,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=0; Total processed=0; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-2:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.19.0.3:11111:291748221*stg/12/0000000c@S0000000c] CurrentlyExecuting=null>
--System*WorkItemGroup:Name=*stg/13/0000000d,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=5; Total processed=5; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-8:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.19.0.3:11111:291748221*stg/13/0000000d@S0000000d] CurrentlyExecuting=null>
--System*WorkItemGroup:Name=*stg/14/0000000e,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=11; Total processed=11; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-11:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.19.0.3:11111:291748221*stg/14/0000000e@S0000000e] CurrentlyExecuting=null>
--System*WorkItemGroup:Name=*stg/15/0000000f,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=18; Total processed=18; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-10:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.19.0.3:11111:291748221*stg/15/0000000f@S0000000f] CurrentlyExecuting=null>
--System*WorkItemGroup:Name=*stg/17/00000011,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=29; Total processed=29; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-9:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.19.0.3:11111:291748221*stg/17/00000011@S00000011] CurrentlyExecuting=Unrecoverable Rejection (info: System.InvalidOperationException: Grain directory is stopping
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.CheckIfShouldForward(GrainId grainId, Int32 hopCount, String operationDescription)
at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount)
at Orleans.Runtime.Scheduler.AsyncClosureWorkItem`1.Execute()
at Orleans.Runtime.Placement.RandomPlacementDirector.OnSelectActivation(PlacementStrategy strategy, GrainId target, IPlacementRuntime context)
at Orleans.Runtime.Placement.PlacementDirectorsManager.SelectOrAddActivation(ActivationAddress sendingAddress, PlacementTarget targetGrain, IPlacementRuntime context, PlacementStrategy strategy)
at Orleans.Runtime.Dispatcher.AddressMessageAsync(Message message, PlacementTarget target, PlacementStrategy strategy, ActivationAddress targetAddress)
at Orleans.Runtime.Dispatcher.<>c__DisplayClass37_0.<<AsyncSendMessage>b__1>d.MoveNext()) Response S172.19.0.3:11111:291748221*grn/13BAB4D8/00000000+OrleansCluster->S172.19.0.3:11111:291748221*stg/17/00000011@S00000011 #9: >
--System*WorkItemGroup:Name=*stg/19/00000013,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=0; Total processed=0; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-12:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.19.0.3:11111:291748221*stg/19/00000013@S00000013] CurrentlyExecuting=null>
--System*WorkItemGroup:Name=*stg/20/00000014,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=10; Total processed=10; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-1:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.19.0.3:11111:291748221*stg/20/00000014@S00000014] CurrentlyExecuting=null>
--System*WorkItemGroup:Name=*stg/22/00000016,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=5; Total processed=5; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-4:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.19.0.3:11111:291748221*stg/22/00000016@S00000016] CurrentlyExecuting=null>
--System*WorkItemGroup:Name=*stg/24/00000018,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=0; Total processed=0; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-7:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.19.0.3:11111:291748221*stg/24/00000018@S00000018] CurrentlyExecuting=null>
--System*WorkItemGroup:Name=*stg/27/0000001b,WorkGroupStatus=Waiting. Currently QueuedWorkItems=0; Total EnQueued=0; Total processed=0; Quantum expirations=0; TaskRunner=ActivationTaskScheduler-3:Queued=0; Detailed SchedulingContext=<[SystemTarget: S172.19.0.3:11111:291748221*stg/27/0000001b@S0000001b] CurrentlyExecuting=null>]
warn: Orleans.Runtime.Silo[100220]
Process is exiting
info: Orleans.Runtime.Silo[100427]
Silo termination is in progress - Will wait for it to finish
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment