Socket.Send freeze under load ?

[3psil0N]

Hello,
I am trying to use this wonderful library to build a simple windows service that receives delivery receipts and inserts them into a database.

I am testing the service using the smppsim application available here:
http://www.seleniumsoftware.com/

I use a simple script to send 1000 messages with bind as transmitter and registered_delivery = 1, and then I bind with the service as receiver and test stability under load (with the simulator).

What consistently happens is that after approximately 600 messages, suddenly the event stops getting fired, and when I try to call unbind from another thread, nothing happens.

I drilled down and found the issue that the following stack:


consistently appears in each time, Socket.Send never returns.
I inspected the bytes and saw that it is never returning from sending the DeliverSmResp PDU.

The issue is reproducible with the demo application as well. Send 1000 messages or so to the simulator and the connect the demo app in receiver mode (on localhost) and it will stop responding after 500-600 messages on my machine. (Windows 7, Core 2 Duo P8600).

What is the best practice for gracefully shutting down when under load ?
When I call unbind() from another thread while the events are being fired, this induces the freeze as well.

Any feedback ? Is this some kind of synchronization issue ? Deadlock ?

[3psil0N]

Here is the smppsim.props file I used to test.

http://pastebin.com/Ep6KRH9U

The threshold seems to be above 1000 messages, around 1200 or so.
It seems to happen more when both the simulator and receiver are on localhost.

[NickAskew]

Is this some kind of synchronization issue ? Deadlock ?

I'd have to say yes, as I haven't experienced the behaviour you mention.
How much work are you doing in the events?

To ensure stability under severe load I adopted the following within my code:

1) Do not perform any work directly on the received data within the raised events.
2) Do not write data from a raised event directly into persistent storage, store it within a list (perform your own synchronisation on the list prior to storage).
3) Create another thread to empty the list and store it in your persistent storage medium (ensuring you perform synchronisation on the list)
This ensures the event buffer is flushed as quickly as possible.
4) Do any work you need to perform on the messages in another thread, taking the data from your persistant storage which is now completely detached from the raised event portion of your application.

[3psil0N]

- All I do is just insert into a database, that's all.

- I expect ALT.SMS to be receiving in asynchronous mode, ie: it should not block on my code.

- I already suspected this, and retried the test several times where the event did nothing, and it still hung with the simulator.

- Also while this is happening, let's say you got the scenario where you need to shut down even under load, so I start rejecting everything coming with queue full to give a chance for the other thread to send its unbind, but then the unbind thread hangs on the same stack. (and in the simulator log, the response seems to never reach it).

- Unbind Socket.Send doesn't return when receiving large numbers of incoming messages.

- Anyway, I got fed up and ported several java libraries to C# (no decent C# library available, RoaminSMPP is a pile of spaghetti and full of bugs).

- I tried OpenSMPP (http://sourceforge.net/apps/trac/smstools/wiki) which exhibited exactly the same behavior, and finally I settled on jsmpp
(http://code.google.com/p/jsmpp/) which passed my simulator tests.

- It properly spawns multiple processing threads on received PDUs and doesn't hang or exhibit any strange behavior under load. Even its unbind works properly even in the face of huge simultaneous incoming messages.

- For the curious, I ported it using ikvm tools (IKVMC). It ports properly, but you have to exclude the unit test classes and some problematic parts of log4j (JMS, and SMTP appenders) that didn't port correctly.

- One downside is that it doesn't implement any encodings (ie: it just gives you a byte array and by default interprets short_message in the default platform encoding which is wrong), but by implementing my own Gsm338Encoding class it works fine.

- I will reconsider alt.sms in the future when a new version is released.

[alt]

Hello 3psil0N,

New version already released. Sorry, I had no time to publish it.
In this version deadlock should be fixed. Old version had bug in receive callback.

[3psil0N]

I tested with this new version on Selenium simulator (SmppSim).

The behavior is a little changed, now instead of freezing forever, it stops for 20 seconds or so and then the Error event fires with an exception of Socket Timed out, and then the unbind also fails because now the socket is closed.

Is this the normal behavior ?