{{Quickfixn}} Socket Deadlock Issue

[Matt Wood]

I agree. If creating a different thread for processing methods is what
the developer *should do*, then it should say so in the
documentation/example apps. The example apps (since I last used them)
make no mention of the dangers of spending too much time within the
processing thread. One or the other: documentation or protection.

-Matt

On Fri, Jun 8, 2012 at 11:03 AM, Christian Jungers
<christian.jungers at cm3.com> wrote:
> Right … the issue is that this relies on the application developer to “do
> the right thing” and keep their receive thread separate from their
> processing thread. But even if the application does use this pattern, the
> problem I’m seeing right now is that the internal QuickFIX.NET library
> doesn’t follow that pattern. Thus, with the large resend request messages
> being automatically / immediately processed on the receive thread, it opens
> the FIX engine up to the possibility of a blocked socket.
>
> So all I’m suggesting is to take that pattern and apply it internally to
> QuickFIX as well. That way, whether or not the application developer follows
> this pattern, they will be protected by a robust and scalable QuickFIX
> engine.
>
> - Christian
>
>
>
> Christian.Jungers at CM3.com - Chief Technology Officer - Tel
>  877.263.1669 x705 - Fax 877.263.1669
>
>
>
> On Fri, Jun 8, 2012 at 10:56 AM, Thomas Tomiczek <t.tomiczek at nettecture.com>
> wrote:
>>
>> But is this not a bad pattern?
>>
>>
>>
>> The way I handle this is:
>>
>>
>>
>> ·         Take incoming message
>>
>> ·         Propagate to my own queue, then handle it from there.
>>
>>
>>
>> As a result, I am not even “cracking” messages in the receiving thread,
>> but later in the processing handler. This one runs, as I said, in a
>> separate thread with an in memory queue ;)
>>
>>
>>
>> Yes, it COULD fill up, but that would require a large amount of messages
>> ;)
>>
>>
>>
>> Regards
>>
>>
>>
>> Thomas
>>
>>
>>
>> From: quickfixn-bounces at lists.quickfixn.com
>> [mailto:quickfixn-bounces at lists.quickfixn.com] On Behalf Of Matt Wood
>> Sent: 08 June 2012 16:31
>> To: Mailing list for QuickFIX/n
>> Subject: Re: {{Quickfixn}} Socket Deadlock Issue
>>
>>
>>
>> Christian,
>>
>>
>>
>> Hi, I'm another user and recent contributor of the quickfix/n library. I
>> have encountered this in a related fashion (but triggered differently) and
>> agree with your final paragraph:
>>
>>
>>
>> "However, in the long run, we may need / want to have distinct receive and
>> process threads. As it is, the same thread will block while calling into the
>> actual FIX application for message processing. Should that processing take a
>> long time or involve, for whatever reason, sending a large number of
>> messages ... then we could wind up in a similar problem. Say the FIX
>> application took 2 minutes to process some particular received message (in a
>> blocking synchronous way) ... and during that 2 minutes, the other side of
>> the FIX connection had sent enough messages to fill up it's socket buffer
>> and block. By ensuring that message processing is in a separate thread from
>> the socket reading then we will guarantee (in a far better way) that our
>> socket should never wind up inadvertently blocking the other side. "
>>
>>
>>
>> Some of my processing (mainly database updates) of messages causes a pile
>> up when receiving an exceptionally large number of messages in a short time
>> period. Specifically, I get this at the end of the day (around 5pm) when the
>> system we're connected to propagates a series of IOI messages to mark market
>> offerings as being unavailable (due to traders being forcibly logged out of
>> the system for the day). The pile up manifests itself as "TCP ZeroWindow"
>> messages to be sent from our side telling the foreign system to delay
>> sending further messages. Eventually the delayed TCP messages are sent and
>> received as our system catches up, but are rejected by our fix engine due to
>> the fact that the messages exceed the configured latency (2 minutes!). Of
>> course there is much I can do to alleviate this particular problem, such as
>> creating some sort of queue and threading at a higher place in the app.
>> However, I do feel that a distinct processing thread makes sense and would
>> benefit the overall robustness of the application.
>>
>>
>>
>> -Matt Wood
>>
>>
>>
>>
>> _______________________________________________
>> Quickfixn mailing list
>> Quickfixn at lists.quickfixn.com
>> http://lists.quickfixn.com/listinfo.cgi/quickfixn-quickfixn.com
>>
>
>
> _______________________________________________
> Quickfixn mailing list
> Quickfixn at lists.quickfixn.com
> http://lists.quickfixn.com/listinfo.cgi/quickfixn-quickfixn.com
>