{{Quickfixn}} Socket Deadlock Issue

[Christian Jungers]

However, I'd point out again that just telling developers to do the right
thing still won't protect QuickFIX from itself ... as in my context
(ResendRequest).
*
*
*
*
*
*
*Christian.Jungers at CM3.com - Chief Technology Officer** - Tel
877.263.1669 x705
- Fax 877.263.1669*



On Fri, Jun 8, 2012 at 11:36 AM, Matt Wood <mjwood7 at gmail.com> wrote:

> I agree. If creating a different thread for processing methods is what
> the developer *should do*, then it should say so in the
> documentation/example apps. The example apps (since I last used them)
> make no mention of the dangers of spending too much time within the
> processing thread. One or the other: documentation or protection.
>
> -Matt
>
> On Fri, Jun 8, 2012 at 11:03 AM, Christian Jungers
> <christian.jungers at cm3.com> wrote:
> > Right … the issue is that this relies on the application developer to “do
> > the right thing” and keep their receive thread separate from their
> > processing thread. But even if the application does use this pattern, the
> > problem I’m seeing right now is that the internal QuickFIX.NET library
> > doesn’t follow that pattern. Thus, with the large resend request messages
> > being automatically / immediately processed on the receive thread, it
> opens
> > the FIX engine up to the possibility of a blocked socket.
> >
> > So all I’m suggesting is to take that pattern and apply it internally to
> > QuickFIX as well. That way, whether or not the application developer
> follows
> > this pattern, they will be protected by a robust and scalable QuickFIX
> > engine.
> >
> > - Christian
> >
> >
> >
> > Christian.Jungers at CM3.com - Chief Technology Officer - Tel
> >  877.263.1669 x705 - Fax 877.263.1669
> >
> >
> >
> > On Fri, Jun 8, 2012 at 10:56 AM, Thomas Tomiczek <
> t.tomiczek at nettecture.com>
> > wrote:
> >>
> >> But is this not a bad pattern?
> >>
> >>
> >>
> >> The way I handle this is:
> >>
> >>
> >>
> >> ·         Take incoming message
> >>
> >> ·         Propagate to my own queue, then handle it from there.
> >>
> >>
> >>
> >> As a result, I am not even “cracking” messages in the receiving thread,
> >> but later in the processing handler. This one runs, as I said, in a
> >> separate thread with an in memory queue ;)
> >>
> >>
> >>
> >> Yes, it COULD fill up, but that would require a large amount of messages
> >> ;)
> >>
> >>
> >>
> >> Regards
> >>
> >>
> >>
> >> Thomas
> >>
> >>
> >>
> >> From: quickfixn-bounces at lists.quickfixn.com
> >> [mailto:quickfixn-bounces at lists.quickfixn.com] On Behalf Of Matt Wood
> >> Sent: 08 June 2012 16:31
> >> To: Mailing list for QuickFIX/n
> >> Subject: Re: {{Quickfixn}} Socket Deadlock Issue
> >>
> >>
> >>
> >> Christian,
> >>
> >>
> >>
> >> Hi, I'm another user and recent contributor of the quickfix/n library. I
> >> have encountered this in a related fashion (but triggered differently)
> and
> >> agree with your final paragraph:
> >>
> >>
> >>
> >> "However, in the long run, we may need / want to have distinct receive
> and
> >> process threads. As it is, the same thread will block while calling
> into the
> >> actual FIX application for message processing. Should that processing
> take a
> >> long time or involve, for whatever reason, sending a large number of
> >> messages ... then we could wind up in a similar problem. Say the FIX
> >> application took 2 minutes to process some particular received message
> (in a
> >> blocking synchronous way) ... and during that 2 minutes, the other side
> of
> >> the FIX connection had sent enough messages to fill up it's socket
> buffer
> >> and block. By ensuring that message processing is in a separate thread
> from
> >> the socket reading then we will guarantee (in a far better way) that our
> >> socket should never wind up inadvertently blocking the other side. "
> >>
> >>
> >>
> >> Some of my processing (mainly database updates) of messages causes a
> pile
> >> up when receiving an exceptionally large number of messages in a short
> time
> >> period. Specifically, I get this at the end of the day (around 5pm)
> when the
> >> system we're connected to propagates a series of IOI messages to mark
> market
> >> offerings as being unavailable (due to traders being forcibly logged
> out of
> >> the system for the day). The pile up manifests itself as "TCP
> ZeroWindow"
> >> messages to be sent from our side telling the foreign system to delay
> >> sending further messages. Eventually the delayed TCP messages are sent
> and
> >> received as our system catches up, but are rejected by our fix engine
> due to
> >> the fact that the messages exceed the configured latency (2 minutes!).
> Of
> >> course there is much I can do to alleviate this particular problem,
> such as
> >> creating some sort of queue and threading at a higher place in the app.
> >> However, I do feel that a distinct processing thread makes sense and
> would
> >> benefit the overall robustness of the application.
> >>
> >>
> >>
> >> -Matt Wood
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> Quickfixn mailing list
> >> Quickfixn at lists.quickfixn.com
> >> http://lists.quickfixn.com/listinfo.cgi/quickfixn-quickfixn.com
> >>
> >
> >
> > _______________________________________________
> > Quickfixn mailing list
> > Quickfixn at lists.quickfixn.com
> > http://lists.quickfixn.com/listinfo.cgi/quickfixn-quickfixn.com
> >
> _______________________________________________
> Quickfixn mailing list
> Quickfixn at lists.quickfixn.com
> http://lists.quickfixn.com/listinfo.cgi/quickfixn-quickfixn.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.quickfixn.com/pipermail/quickfixn-quickfixn.com/attachments/20120608/c20900ae/attachment-0002.htm>