evFullMessageReceived Error in v2.9.0-beta-2211

Post Reply
vineet
Posts: 28
Joined: Sat May 06, 2017 7:06 pm

evFullMessageReceived Error in v2.9.0-beta-2211

Post by vineet » Mon Nov 23, 2020 9:32 am

Hi alt,

Almost everything is working fine in the upgrade to v2 only some few errors are showing in log occasionally. If you can please assist us why these errors are coming.

1. Event evFullMessageReceived failed Exception: System.ArgumentNullException: Array cannot be null. Parameter name: bytes at System.Text.Encoding.GetString(Byte[] bytes) at Inetlab.SMPP.Common.MessageComposer.OnFullMessageReceived[TSmppMessage](TSmppMessage message). (SMPP Server MessageComposerObject.evFullMessageReceived += OnFullMessageReceived Event)

This is coming very few times but i think it is internal library error. can you describe what could be the reason. Is it because of the client sending some wrong pdu or something else, can you explain possible cases when this error might come up.

2. UserData cannot be longer than 255 bytes. Current length is 266.; InnerException :; StackTrace : at Inetlab.SMPP.Common.SmppWriter.WriteUserData(BufferWriter writer, UserData data) at Inetlab.SMPP.Common.SmppWriter.Write(DeliverSm pdu) at Inetlab.SMPP.PDU.PduExtension.Serialize(SmppPDU pdu, EncodingMapper encodingMapper) at SMPP_Node_V3.SMSSender.client_evDeliverSm(Object sender, DeliverSm data)

We are getting this error from only one vendor we have opened up a Ticket with them can you please explain why this error is coming in library so we can resolve this with vendor.

3. I see you have set in the demo app SMPPClient.WorkerThreads = 3; (Both client and ServerClient)
First thing what is the use of the property WorkerThreads if we set this property to 1 is there will be any issue. As far we understand it will start 3 threads for processing on each bind.
Currently we have around 900 bind (both client and smppserver client) on our live server so the number of thread created is very high 900 * 3 = 2700
What is the logic behind setting no of threads what factors we should consider setting up the number of threads on each bind. is there something we need to look at like TPS, or the volume of sms is sending on each bind so we can set this value
appropriately.

4. 95.216.74.23:52478 => InterNetwork/112.196.55.205:10218 Read data failed Exception: Inetlab.SMPP.Common.SmppReaderException: Packet length 1949975621 is longer than 1Mb. at Inetlab.SMPP.Common.SmppSocket.ReadData() at Inetlab.SMPP.Common.SmppSocket.<ReadDataFromNetworkAsync>d__75.MoveNext();

This error comes up whenever we send lots of messages in bulk. then all messages comes up with smppclient_noconnection error.

That is all the issues we are facing for now an and everything else seems stable now.

Thanks
alt
Site Admin
Posts: 985
Joined: Tue Apr 25, 2006 9:45 am

Re: evFullMessageReceived Error in v2.9.0-beta-2211

Post by alt » Mon Nov 23, 2020 8:50 pm

Hi vineet,
vineet wrote:
Mon Nov 23, 2020 9:32 am
1. Event evFullMessageReceived failed Exception: System.ArgumentNullException: Array cannot be null. Parameter name: bytes at System.Text.Encoding.GetString(Byte[] bytes) at Inetlab.SMPP.Common.MessageComposer.OnFullMessageReceived[TSmppMessage](TSmppMessage message). (SMPP Server MessageComposerObject.evFullMessageReceived += OnFullMessageReceived Event)
I've reproduced this issue. It happens when you receive a PDU with empty short_message and message_payload. The fix is on the way.
vineet wrote:
Mon Nov 23, 2020 9:32 am
2. UserData cannot be longer than 255 bytes. Current length is 266.; InnerException :; StackTrace : at Inetlab.SMPP.Common.SmppWriter.WriteUserData(BufferWriter writer, UserData data) at Inetlab.SMPP.Common.SmppWriter.Write(DeliverSm pdu) at Inetlab.SMPP.PDU.PduExtension.Serialize(SmppPDU pdu, EncodingMapper encodingMapper) at SMPP_Node_V3.SMSSender.client_evDeliverSm(Object sender, DeliverSm data)
Could you send me a dump of this PDU?
vineet wrote:
Mon Nov 23, 2020 9:32 am
3. I see you have set in the demo app SMPPClient.WorkerThreads = 3; (Both client and ServerClient)
SMPPClient.WorkerThreads defines how many SMPP requests from a client your application can process at the same time. It is old solution from v1. This was supposed to help process requests from the client faster and reduce the impact of traffic from other connected clients. In some cases, it is better to handle requests in a ThreadPool, but for this the application must be optimized so that the handler does not take up the ThreadPool-thread for a long time. I plan to provide an ability to change receiver TaskScheduler for the SmppClientBase.
vineet wrote:
Mon Nov 23, 2020 9:32 am
4. 95.216.74.23:52478 => InterNetwork/112.196.55.205:10218 Read data failed Exception: Inetlab.SMPP.Common.SmppReaderException: Packet length 1949975621 is longer than 1Mb. at Inetlab.SMPP.Common.SmppSocket.ReadData() at Inetlab.SMPP.Common.SmppSocket.<ReadDataFromNetworkAsync>d__75.MoveNext();


I think the SMPP stream from provider was corrupted. In some point of time it starts to send inconsistent data. The library detects this situation and drops the connection. I would suggest to analyze SMPP traffic with Wireshark.
vineet
Posts: 28
Joined: Sat May 06, 2017 7:06 pm

Re: evFullMessageReceived Error in v2.9.0-beta-2211

Post by vineet » Tue Nov 24, 2020 5:28 am

Hi alt,
I think the SMPP stream from provider was corrupted. In some point of time it starts to send inconsistent data. The library detects this situation and drops the connection. I would suggest to analyze SMPP traffic with Wireshark.
There is no issue while sending a batch of 120000 Messages at once in V1 library. but when ever we switch to v2 library the vendor gets disconnected just after 1500-2000 submissions and rest of all the messages goes in SMPP_NOCONN error. please provide a solution for this because of this major issue we not able to implement it on live server.

We are not able to debug this issue either tried all we can. but in log we see that vendor connection never gets reconnected its always giving like shutdown send. client disconnected. in logs. and in bind response it give smpp_noconn how can it give smpp_noconn in bind when the connectasync return true.
SMPPClient.WorkerThreads defines how many SMPP requests from a client your application can process at the same time. It is old solution from v1. This was supposed to help process requests from the client faster and reduce the impact of traffic from other connected clients. In some cases, it is better to handle requests in a ThreadPool, but for this the application must be optimized so that the handler does not take up the ThreadPool-thread for a long time. I plan to provide an ability to change receiver TaskScheduler for the SmppClientBase.

So it must not be less than 3 worker threads.

Could you send me a dump of this PDU?
How to save pdu dump to file.
alt
Site Admin
Posts: 985
Joined: Tue Apr 25, 2006 9:45 am

Re: evFullMessageReceived Error in v2.9.0-beta-2211

Post by alt » Wed Nov 25, 2020 8:41 pm

vineet wrote:
Tue Nov 24, 2020 5:28 am
There is no issue while sending a batch of 120000 Messages at once in V1 library. but when ever we switch to v2 library the vendor gets disconnected just after 1500-2000 submissions and rest of all the messages goes in SMPP_NOCONN error. please provide a solution for this because of this major issue we not able to implement it on live server.
Do you see any warnings or error in the library log with V1?
The reading from network is very simple: Each SMPP PDU begins with 4 bytes of the length with following bytes of data. Library uses synchronous reading in one thread for a SMPP session, so multi-thread issues are impossible. If remote side sends wrong PDU length not matching data bytes the SMPP stream is considered as corrupted. And library drops connection and the application receives responses with NOCONN status.
It is not that hard to change the read of the network stream so that it looks for the next correct PDU. But I think that this is the wrong way, which leads to unpredictable behavior and difficulties in analyzing the problems.
vineet wrote:
Tue Nov 24, 2020 5:28 am
We are not able to debug this issue either tried all we can. but in log we see that vendor connection never gets reconnected its always giving like shutdown send. client disconnected. in logs. and in bind response it give smpp_noconn how can it give smpp_noconn in bind when the connectasync return true.
Is reconnect enabled for this client? Do you expect that library sends such PDU again when it binds again?

Can you enable debug log for this case and send me that part where library disconnects and tries to reconnect.
https://docs.inetlab.com/smpp/v2.8/arti ... gging.html
vineet wrote:
Tue Nov 24, 2020 5:28 am
How to save pdu dump to file.
alt wrote:
Mon Nov 23, 2020 8:50 pm
UserData cannot be longer than 255 bytes. Current length is 266.; InnerException :; StackTrace : at Inetlab.SMPP.Common.SmppWriter.WriteUserData(BufferWriter writer, UserData data) at Inetlab.SMPP.Common.SmppWriter.Write(DeliverSm pdu) at Inetlab.SMPP.PDU.PduExtension.Serialize(SmppPDU pdu, EncodingMapper encodingMapper) at SMPP_Node_V3.SMSSender.client_evDeliverSm(Object sender, DeliverSm data)
I suppose you received a delivery receipt in message_payload parameter. Could you enable Verbose log level for the library and send me received byte array for this message? The wireshark capture file can also help in this case.
vineet
Posts: 28
Joined: Sat May 06, 2017 7:06 pm

Re: evFullMessageReceived Error in v2.9.0-beta-2211

Post by vineet » Thu Nov 26, 2020 7:57 am

Do you see any warnings or error in the library log with V1?
The reading from network is very simple: Each SMPP PDU begins with 4 bytes of the length with following bytes of data. Library uses synchronous reading in one thread for a SMPP session, so multi-thread issues are impossible. If remote side sends wrong PDU length not matching data bytes the SMPP stream is considered as corrupted. And library drops connection and the application receives responses with NOCONN status.
It is not that hard to change the read of the network stream so that it looks for the next correct PDU. But I think that this is the wrong way, which leads to unpredictable behavior and difficulties in analyzing the problems.


Is reconnect enabled for this client? Do you expect that library sends such PDU again when it binds again?
No Reconnect Is Not Enabled. We check for re connection manually after 1 minute.

Can you enable debug log for this case and send me that part where library disconnects and tries to reconnect.
https://docs.inetlab.com/smpp/v2.8/arti ... gging.html

I suppose you received a delivery receipt in message_payload parameter. Could you enable Verbose log level for the library and send me received byte array for this message? The wireshark capture file can also help in this case.
See the attached wireshark file and check for phonenumber : 917889570141 DeliverSM. we got this error on this Deliversm only not all deliversm sent by this vendor. In this wireshark capture there is only one i can find which gave us this error. i hope it will help you.

UPDATE: alt we are also getting this same error (Deliversm Serialization) in latest v1 library. can you also help in this.

New Error Found In Logs During DeliverSM serialization
Input string was not in a correct format.; InnerException :; StackTrace : at System.Number.StringToNumber(String str, NumberStyles options, NumberBuffer& number, NumberFormatInfo info, Boolean parseDecimal) at System.Number.ParseUInt32(String value, NumberStyles options, NumberFormatInfo numfmt) at System.UInt16.Parse(String s, NumberStyles style, NumberFormatInfo info) at Inetlab.SMPP.PDU.DeliverSm.ToSmppBuffer() at Inetlab.SMPP.PDU.DeliverSm.GetBytes() at SMPP_Service_V2.SMSSender.client_evDeliverSm(Object sender, DeliverSm data)
Attachments
VCON.zip
(8.5 KiB) Downloaded 402 times
alt
Site Admin
Posts: 985
Joined: Tue Apr 25, 2006 9:45 am

Re: evFullMessageReceived Error in v2.9.0-beta-2211

Post by alt » Sun Nov 29, 2020 4:22 pm

vineet wrote:
Thu Nov 26, 2020 7:57 am
See the attached wireshark file and check for phonenumber : 917889570141 DeliverSM. we got this error on this Deliversm only not all deliversm sent by this vendor. In this wireshark capture there is only one i can find which gave us this error. i hope it will help you.
The DeliverSm PDU with sequence number 2303 has wrong data_coding. The short_message field is encoded in ASCII, but specified UCS2 (0x08). Even Wireshark shows unreadable content. The real content of short_message field is:
id:S21810218261303255494130253 sub:001 dlvrd:001 submit date:201126130325 done date:201126130327 stat:DELIVRD err:000 text:Your payment of 6

I have improved delivery receipt parsing. Library tries to work around this issue when ASCII text can be decoded. Please test latest 2.9-beta from today.
vineet
Posts: 28
Joined: Sat May 06, 2017 7:06 pm

Re: evFullMessageReceived Error in v2.9.0-beta-2211

Post by vineet » Mon Nov 30, 2020 5:42 am

I have improved delivery receipt parsing. Library tries to work around this issue when ASCII text can be decoded. Please test latest 2.9-beta from today.
Thanks for the update in v2. but we are still not confident about the v2 so we cannot go live with it right now and test it. can you please provide this same fix for the version 1.3.2 also so we can test it on live.

Thanks
alt
Site Admin
Posts: 985
Joined: Tue Apr 25, 2006 9:45 am

Re: evFullMessageReceived Error in v2.9.0-beta-2211

Post by alt » Mon Nov 30, 2020 4:54 pm

The version 1.x is not supported anymore.
alt
Site Admin
Posts: 985
Joined: Tue Apr 25, 2006 9:45 am

Re: evFullMessageReceived Error in v2.9.0-beta-2211

Post by alt » Mon Nov 30, 2020 7:43 pm

vineet wrote:
Mon Nov 30, 2020 5:42 am
we are still not confident about the v2 so we cannot go live with it right now and test it.
What is the reason of it? Do you mean the issue with corrupted SMPP stream from remote side?
vineet
Posts: 28
Joined: Sat May 06, 2017 7:06 pm

Re: evFullMessageReceived Error in v2.9.0-beta-2211

Post by vineet » Tue Dec 01, 2020 11:21 am

alt wrote:
Mon Nov 30, 2020 7:43 pm
vineet wrote:
Mon Nov 30, 2020 5:42 am
we are still not confident about the v2 so we cannot go live with it right now and test it.
What is the reason of it? Do you mean the issue with corrupted SMPP stream from remote side?
Hi alt, The main reason is the stability of the v2 version. I don't know what happens but when v2 is put under load is starts behaving weirdly and giving many errors. We have a daily traffic of 4-5 millions messages with customer base of over 10k+.
Whenever we switch to v2 we lose at least an hour of traffic because of the issues then we have to switch back to v1. I already told you v2 behaves as it should be in local test environment also on pre production it works fine but when the actual loads comes with 500+ vendor binds and 1000+ SMPP Server Client Connections this library is not able to handle this but v1 creates no issues.

If you can suggest a better way to load test on pre production it would be great.

As we really want to upgrade but stability is our main concern.

Thanks
alt
Site Admin
Posts: 985
Joined: Tue Apr 25, 2006 9:45 am

Re: evFullMessageReceived Error in v2.9.0-beta-2211

Post by alt » Tue Dec 01, 2020 7:58 pm

I would like to know all exceptions you got in production system. When it is possible with full call stack.

From the list above I see only one error that was not fixed from your point of view: SmppReaderException: Packet length 1949975621 is longer than 1Mb.
Does this error appears only with one certain endpoint or it is related many others?
I asked you to check if you get any warnings/errors in log with this endpoint when you send messages in bulk using v1. When the same issue exists, library should write log message like "Unknown data in stream. Length:".

Could you send me another errors your are facing?
alt
Site Admin
Posts: 985
Joined: Tue Apr 25, 2006 9:45 am

Re: evFullMessageReceived Error in v2.9.0-beta-2211

Post by alt » Mon Dec 28, 2020 9:32 pm

Hi vineet,

I would like to ask you to test the version 2.9.0-beta-2249.
I did stress test for this version and made some improvements.
Please check ChangeLog for details.
Post Reply