<div>Right … the issue is that this relies on the application developer to “do the right thing” and keep their receive thread separate from their processing thread. But even if the application does use this pattern, the problem I’m seeing right now is that the internal QuickFIX.NET library doesn’t follow that pattern. Thus, with the large resend request messages being automatically / immediately processed on the receive thread, it opens the FIX engine up to the possibility of a blocked socket.</div>
<div><br></div><div>So all I’m suggesting is to take that pattern and apply it internally to QuickFIX as well. That way, whether or not the application developer follows this pattern, they will be protected by a robust and scalable QuickFIX engine.</div>
<div><br></div><div><span class="Apple-tab-span" style="white-space:pre"> </span>- Christian</div><span style="border-collapse:collapse"><font><div style="font-weight:bold;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px">
<span style="border-collapse:collapse;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px"><font face="verdana, sans-serif" size="1"><b><br></b></font></span></div><div style="font-weight:bold;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px">
<span style="border-collapse:collapse;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px"><font face="verdana, sans-serif" size="1"><b><br></b></font></span></div><div style="font-weight:bold;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px">
<span style="border-collapse:collapse;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px"><font face="verdana, sans-serif" size="1"><b><br></b></font></span></div><div style="font-weight:bold;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px">
<span style="border-collapse:collapse;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px"><font face="verdana, sans-serif" size="1"><b>Christian.Jungers@CM3.com - <span style="font-family:arial,sans-serif;font-size:13px;font-weight:normal"><b><span style="font-size:7pt;color:rgb(0,51,102)">Chief Technology Officer</span></b></span></b></font></span><b><span style="font-family:arial,sans-serif;font-size:13px;font-weight:normal"><b><span style="font-size:7pt;color:rgb(0,51,102)"><span style="color:rgb(136,136,136);font-family:Verdana,sans-serif;font-weight:normal"><span style="color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px;font-weight:bold"><b><span style="font-family:arial,sans-serif;font-size:13px;font-weight:normal"><b><span style="font-size:7pt;color:rgb(0,51,102)"><span style="color:rgb(136,136,136);font-family:Verdana,sans-serif;font-weight:normal"> </span></span></b></span></b><span style="font-family:verdana,sans-serif;font-size:x-small"><b>- </b></span></span></span></span></b></span><span style="font-family:verdana,sans-serif;font-size:x-small;font-weight:normal">Tel <a style="color:rgb(0,101,204)">877.263.1669</a> x705 - </span><span style="font-family:verdana,sans-serif;font-size:x-small;font-weight:normal">Fax <a style="color:rgb(0,101,204)">877.263.1669</a></span></b></div>
</font></span><br>
<br><br><div class="gmail_quote">On Fri, Jun 8, 2012 at 10:56 AM, Thomas Tomiczek <span dir="ltr"><<a href="mailto:t.tomiczek@nettecture.com" target="_blank">t.tomiczek@nettecture.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-GB" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">But is this not a bad pattern?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">The way I handle this is:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p><u></u><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Take incoming message<u></u><u></u></span></p>
<p><u></u><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Propagate to my own queue, then handle it from there.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">As a result, I am not even “cracking” messages in the receiving thread, but later in the processing handler. This one runs, as I said, in a separate thread
with an in memory queue ;)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Yes, it COULD fill up, but that would require a large amount of messages ;)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Regards<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Thomas<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> <a href="mailto:quickfixn-bounces@lists.quickfixn.com" target="_blank">quickfixn-bounces@lists.quickfixn.com</a> [mailto:<a href="mailto:quickfixn-bounces@lists.quickfixn.com" target="_blank">quickfixn-bounces@lists.quickfixn.com</a>]
<b>On Behalf Of </b>Matt Wood<br>
<b>Sent:</b> 08 June 2012 16:31<br>
<b>To:</b> Mailing list for QuickFIX/n<br>
<b>Subject:</b> Re: {{Quickfixn}} Socket Deadlock Issue<u></u><u></u></span></p><div class="im">
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Christian,<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Hi, I'm another user and recent contributor of the quickfix/n library. I have encountered this in a related fashion (but triggered differently) and agree with your final paragraph:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">"However, in the long run, we may need / want to have distinct receive and process threads. As it is, the same thread will block while calling into the actual FIX application for message processing. Should that processing take a long time
or involve, for whatever reason, sending a large number of messages ... then we could wind up in a similar problem. Say the FIX application took 2 minutes to process some particular received message (in a blocking synchronous way) ... and during that 2 minutes,
the other side of the FIX connection had sent enough messages to fill up it's socket buffer and block. By ensuring that message processing is in a separate thread from the socket reading then we will guarantee (in a far better way) that our socket should never
wind up inadvertently blocking the other side. "<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Some of my processing (mainly database updates) of messages causes a pile up when receiving an exceptionally large number of messages in a short time period. Specifically, I get this at the end of the day (around 5pm) when the system we're
connected to propagates a series of IOI messages to mark market offerings as being unavailable (due to traders being forcibly logged out of the system for the day). The pile up manifests itself as "TCP ZeroWindow" messages to be sent from our side telling
the foreign system to delay sending further messages. Eventually the delayed TCP messages are sent and received as our system catches up, but are rejected by our fix engine due to the fact that the messages exceed the configured latency (2 minutes!). Of course
there is much I can do to alleviate this particular problem, such as creating some sort of queue and threading at a higher place in the app. However, I do feel that a distinct processing thread makes sense and would benefit the overall robustness of the application.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">-Matt Wood<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div></div>
</div>
<br>_______________________________________________<br>
Quickfixn mailing list<br>
<a href="mailto:Quickfixn@lists.quickfixn.com">Quickfixn@lists.quickfixn.com</a><br>
<a href="http://lists.quickfixn.com/listinfo.cgi/quickfixn-quickfixn.com" target="_blank">http://lists.quickfixn.com/listinfo.cgi/quickfixn-quickfixn.com</a><br>
<br></blockquote></div><br>