An exception was thrown by a TaskScheduler

Post Reply
vineet
Posts: 28
Joined: Sat May 06, 2017 7:06 pm

An exception was thrown by a TaskScheduler

Post by vineet » Thu Nov 19, 2020 3:47 pm

Hi ,

while connecting to vendor some of the vendors we are getting this error.

An exception was thrown by a TaskScheduler.

Code: Select all

code snippet.

 bool bSuccess = await SMPPClient.ConnectAsync(VendorAccount.Host, Port);
                    if (bSuccess)
                    {
                        WindowsLogger.WriteSMPPCriticalLog(VendorAccount.VendorAccountID, "Success", "Connected Now Binding SMPP Client For System ID : " + VendorAccount.SystemID, "CRITICAL");
                        await Bind();
                    }
                    else
                    {
                        WindowsLogger.WriteSMPPCriticalLog(VendorAccount.VendorAccountID, "Error", "SMPP Client Connection Failed.", "CRITICAL");
                    }
                }
            }
            catch (Exception ex)
            {
                WindowsLogger.WriteSMPPCriticalLog(VendorAccount.VendorAccountID, "Error", "Error Connecting To Vendor Account. Error : " + ex.Message, "CRITICAL");
            }
it is always going in exception.

In Local Machine I connect 500 bind with smppserverdemo it works fine. but as we go live with 240 real world binds it starts giving all these errors and finally give System.OutOfMemoryException and crashes.

List of error cached in logger:
1. An exception was thrown by a TaskScheduler.
2. Exception of type 'System.OutOfMemoryException' was thrown.
3. Bad Status Returned During Bind BindReceiverResp With Status ESME_RINVBNDSTS
4. Value cannot be null. Parameter name: scheduler

please help on this.

library version 2.9.0.2
alt
Site Admin
Posts: 985
Joined: Tue Apr 25, 2006 9:45 am

Re: An exception was thrown by a TaskScheduler

Post by alt » Thu Nov 19, 2020 6:28 pm

Hi vineet,
vineet wrote:
Thu Nov 19, 2020 3:47 pm
1. An exception was thrown by a TaskScheduler.
4. Value cannot be null. Parameter name: scheduler
Please provide all details for these exceptions.
vineet wrote:
Thu Nov 19, 2020 3:47 pm
2. Exception of type 'System.OutOfMemoryException' was thrown.
Out of memory exception can occur in any part of the code. How many messages do you send in one batch? With the version 2.9-beta you can use client.SendQueueLimit property to limit number of requests waiting for response which limits memory consumption for a smpp connection. It helps also to avoid ESME_RMSGQFUL (Message queue full) response status from the SMPP server.
vineet wrote:
Thu Nov 19, 2020 3:47 pm
3. Bad Status Returned During Bind BindReceiverResp With Status ESME_RINVBNDSTS
Do you check bind response status before starting message sending? I don't know what the ESME_RINVBNDSTS status for receiver means. Maybe provider wants to tell you that you have made to many binds. Please ask them.
vineet
Posts: 28
Joined: Sat May 06, 2017 7:06 pm

Re: An exception was thrown by a TaskScheduler

Post by vineet » Thu Nov 19, 2020 6:45 pm

Hi Alt,

1. An exception was thrown by a TaskScheduler.
2. Value cannot be null. Parameter name: scheduler
I do not have the stack trace for these errors but we have no issue with v1 of your library all vendors connect just fine. we tried to work with v2 of your library when it was just launched and we had faced the similar issue that time also as we are not able to connect with vendors on live server. but on local it works fine.

Image

see this attached image for error.

what i noted after getting many ESME_RINVBNDSTS or BIND_FAIL we start getting error An exception was thrown by a TaskScheduler. and Value cannot be null. Parameter name: scheduler
and also after few minutes later we get Exception of type 'System.OutOfMemoryException' was thrown. sometime in our windows service timer and sometime in vendor connection library.

we really need to upgrade to the version 2 of the library please help.

Kindly have a look at the below code how we do connection with v2. or if you can suggest some changes.

Code: Select all

 protected async Task ConnectAllSMPPServer(int? VendorAccountID = null)
        {
            try
            {
                IList<Task> SenderTasks = new List<Task>();

                IEnumerable<VendorAccountModel> VendorAccounts = JobObject.GetAllVendorSMPPAccounts(VendorAccountID); //get List of smpp account accounts currently around 280

                int VersionNumber = Encryption.VersionControl();
                foreach (VendorAccountModel VendorAccount in VendorAccounts)
                {
                    SMSSender SMSSenderObject = new SMSSender(VendorAccount.VendorAccountName, VendorAccount.VendorAccountID, VendorAccount.ResponseTimeout, VendorAccount.EnquireLinkInterval, VendorAccount.SubmitSpeed, VendorAccount.ErrorCodeLevel, VendorAccount.DelayTime);
                    SMSSenderObject.VendorAccount = VendorAccount;
                    SMSSenderObject.VendorAccount.VendorSenderIDTranslation = VendorSenderIDTranslation.Where(x => x.VendorAccountID.Equals(VendorAccount.VendorAccountID)).AsEnumerable();
                    SMSSenderObject.VendorAccount.VendorReRoute = RerouteDetail.Where(x => x.VendorAccountID.Equals(VendorAccount.VendorAccountID)).AsEnumerable();
                    SMSSenderObject.VendorAccount.VersionNumber = VersionNumber;

                    if (VendorAccount.ConnectionMode == "T")
                    {
                        SenderTasks.Add(SMSSenderObject.ConnectToSMPPServer(ConnectionMode.Transceiver, VendorAccount.Port));
                    }
                    else
                    {
                        SenderTasks.Add(SMSSenderObject.ConnectToSMPPServer(ConnectionMode.Transmitter, VendorAccount.Port));
                    }

                    SMSSenderList.TryAdd(VendorAccount.SMPPStatusID, SMSSenderObject);
                }

                await Task.WhenAll(SenderTasks);
            }
            catch (Exception ex)
            {
                WindowsLogger.WriteSMPPCriticalLog(0, "Error", "Error Occurred In ConnectAllSMPPServer. Error : " + ex.Message, "CRITICAL");
            }
        }
Here is the Connect To SMPP Server code

Code: Select all

public async Task ConnectToSMPPServer(ConnectionMode SMPPMode, int Port)
        {
            try
            {
                AutoConnect = true;

                if (SMPPClient.Status == ConnectionStatus.Closed)
                {
                    SMPPClient.EsmeAddress = new SmeAddress("", (AddressTON)Convert.ToByte(VendorAccount.Addr_TON), (AddressNPI)Convert.ToByte(VendorAccount.Addr_NPI));
                    SMPPClient.SystemType = VendorAccount.SystemType;
                    SMPPModeGlobal = SMPPMode;

                    if (VendorAccount.EnableSecureConnection == false)
                    {
                        SMPPClient.EnabledSslProtocols = SslProtocols.None;
                    }
                    else
                    {
                        switch (VendorAccount.SecureConnectionType)
                        {
                            case "DEFAULT":
                                SMPPClient.EnabledSslProtocols = SslProtocols.Default;
                                break;
                            case "TLS":
                                SMPPClient.EnabledSslProtocols = SslProtocols.Tls;
                                break;
                            case "TLS11":
                                SMPPClient.EnabledSslProtocols = SslProtocols.Tls11;
                                break;
                            case "TLS12":
                                SMPPClient.EnabledSslProtocols = SslProtocols.Tls12;
                                break;
                            case "SSL3":
                                SMPPClient.EnabledSslProtocols = SslProtocols.Default;
                                break;
                            case "SSL2":
                                SMPPClient.EnabledSslProtocols = SslProtocols.Default;
                                break;
                        }
                        SMPPClient.ClientCertificates.Clear();
                        SMPPClient.ClientCertificates.Add(new X509Certificate2(Properties.Settings.Default.CertificateFilePath + VendorAccount.CertificateFile, VendorAccount.CertificateFilePassword));
                    }

                    bool bSuccess = await SMPPClient.ConnectAsync(VendorAccount.Host, Port);
                    if (bSuccess)
                    {
                        WindowsLogger.WriteSMPPCriticalLog(VendorAccount.VendorAccountID, "Success", "Connected Now Binding SMPP Client For System ID : " + VendorAccount.SystemID, "CRITICAL");
                        await Bind();
                    }
                    else
                    {
                        WindowsLogger.WriteSMPPCriticalLog(VendorAccount.VendorAccountID, "Error", "SMPP Client Connection Failed.", "CRITICAL");
                        await Reconnect();
                    }
                }
            }
            catch (Exception ex)
            {
                WindowsLogger.WriteSMPPCriticalLog(VendorAccount.VendorAccountID, "Error", "Error Connecting To Vendor Account. Error : " + ex.Message, "CRITICAL");
            }
        }

Bind Code

Code: Select all

private async Task Bind()
        {
            BindResp resp = await SMPPClient.BindAsync(VendorAccount.SystemID, VendorAccount.Password, ConnectionMode.Receiver);

            switch (resp.Header.Status)
            {
                case CommandStatus.ESME_ROK:
                    WindowsLogger.WriteSMPPCriticalLog(VendorAccount.VendorAccountID, "Success", "Binding Completed, Bind Result System Is " + resp.SystemId + " With Status " + resp.Header.Status.ToString(), "CRITICAL");
                    break;
                default:
                    WindowsLogger.WriteSMPPCriticalLog(VendorAccount.VendorAccountID, "Error", "Bad Status Returned During Bind " + resp.Header.Command.ToString() + " With Status " + resp.Header.Status.ToString(), "CRITICAL");
                    await Disconnect();
                    break;
            }
        }
Disconnect

Code: Select all

 public async Task Disconnect()
        {
            if (SMPPClient.Status == ConnectionStatus.Bound)
            {
                await UnBind();
            }

            if (SMPPClient.Status == ConnectionStatus.Open)
            {
                await SMPPClient.DisconnectAsync();
            }
        }
Unbind

Code: Select all

private async Task UnBind()
        {
            WindowsLogger.WriteSMPPCriticalLog(VendorAccount.VendorAccountID, "Information", "Unbinding SMPP Client", "CRITICAL");
            UnBindResp resp = await SMPPClient.UnbindAsync();

            switch (resp.Header.Status)
            {
                case CommandStatus.ESME_ROK:
                    WindowsLogger.WriteSMPPCriticalLog(VendorAccount.VendorAccountID, "Success", "Unbind Status " + resp.Header.Status, "CRITICAL");
                    break;
                default:
                    WindowsLogger.WriteSMPPCriticalLog(VendorAccount.VendorAccountID, "Error", "Bad Status Returned During Unbind : " + resp.Header.Command + " , Status : " + resp.Header.Status, "CRITICAL");
                    await SMPPClient.DisconnectAsync();
                    break;
            }

        }
Reconnect

Code: Select all

private async Task Reconnect()
        {
             Task.Delay(60000).Wait();

            if (AutoConnect == true)
            {
                await NodeService.ReconnectVendorAccount("SENDER", VendorAccount.SMPPStatusID.ToString());
            }
        }
Initilization of library

Code: Select all

public SMSSender(string VendorAccountName, int VendorAccountID, int ResponseTimeout, int EnquireLinkInterval, int SubmitSpeed, string LogLevel, int DelayTimeInSeconds)
        {
            LogLevel LoggerLogLevel = (LogLevel)Enum.Parse(typeof(LogLevel), LogLevel, true);

            LogManager.SetLoggerFactory(new SMPPLogger(LoggerLogLevel, VendorAccountID));

            AppDomain.CurrentDomain.UnhandledException += (sender, args) =>
            {
                LogManager.GetLogger("AppDomain").Fatal((Exception)args.ExceptionObject, "Unhandled Exception");
            };

            _DelayTimeInSeconds = DelayTimeInSeconds;

            SMPPClient = new SmppClient();

            if (DelayTimeInSeconds != 0)
            {
                SMPPClient.SendSpeedLimit = new LimitRate(1, TimeSpan.FromSeconds(DelayTimeInSeconds));
            }
            else
            {
                SMPPClient.SendSpeedLimit = new LimitRate(SubmitSpeed, TimeSpan.FromSeconds(1));
            }

            SMPPClient.EnabledSslProtocols = SslProtocols.None;
            SMPPClient.evDisconnected += new DisconnectedEventHandler(client_evDisconnected);
            SMPPClient.evDeliverSm += new DeliverSmEventHandler(client_evDeliverSm);
            SMPPClient.evUnBind += new UnBindEventHandler(client_evUnBind);
            SMPPClient.evServerCertificateValidation += OnCertificateValidation;
            SMPPClient.evRecoverySucceeded += client_evRecoverySucceed;
            
            SMPPClient.ResponseTimeout = TimeSpan.FromSeconds(ResponseTimeout);
            SMPPClient.EnquireLinkInterval = TimeSpan.FromSeconds(EnquireLinkInterval);
            SMPPClient.ConnectionTimeout = TimeSpan.FromSeconds(30);
            SMPPClient.ConnectionRecovery = false;
            SMPPClient.ConnectionRecoveryDelay = TimeSpan.FromMinutes(2);
            SMPPClient.WorkerThreads = 3;
        }
Please have a look and suggest if any changes possible to get it working on live server.

Thanks
Attachments
Capture.PNG
Capture.PNG (63.36 KiB) Viewed 5097 times
alt
Site Admin
Posts: 985
Joined: Tue Apr 25, 2006 9:45 am

Re: An exception was thrown by a TaskScheduler

Post by alt » Thu Nov 19, 2020 7:30 pm

Please don't use .Wait() method or .Result property for async methods

Task.Delay(60000).Wait();

replace with

await Task.Delay(60000);

I suggest to write in log all exception details with exception.ToString() instead of exception.Message;
Post Reply