<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:116216646;
mso-list-type:hybrid;
mso-list-template-ids:-1043429782 -876605216 134807555 134807557 134807553 134807555 134807557 134807553 134807555 134807557;}
@list l0:level1
{mso-level-start-at:200;
mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">What I do is:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-size:11.0pt;font-family:Symbol;color:#1F497D"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Event driven queue spawning a worker thread when data arrives and non is spawned.<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-size:11.0pt;font-family:Symbol;color:#1F497D"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Using efficient insert mechanisms. Sorry to say, but I am managing 75000 new data activity entries per second from the write loop, and that is on
lower end hardware, limited by the database server IO. Not sure what you use as Database, but for SqlServer do NOT generate INSERT statements, use SqlBulkCopy, in large increments (I think I use 50.000 rows per insert now).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">There should not be processing of any incoming data just for recording, and definitely NOT in the database. – sorry. But that is definitely not at all a QuickFix
question anymore.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Just as note – I have no problem most of the time keeping up with the complete CME group stream (which is around 200.000 symbols, though most very rarely active).
With most capacity left.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Regards<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thomas<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> quickfixn-bounces@lists.quickfixn.com [mailto:quickfixn-bounces@lists.quickfixn.com]
<b>On Behalf Of </b>Ondra Kvet<br>
<b>Sent:</b> 17 July 2012 16:08<br>
<b>To:</b> Mailing list for QuickFIX/n<br>
<b>Subject:</b> Re: {{Quickfixn}} Reconnecting problem<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi everyone, <o:p></o:p></p>
<div>
<p class="MsoNormal">I have found another problem relating to this. I am simply asking the provider for market data and saving them to database. I implemented Grant's solution in following way:
<br>
<br>
<i><span style="color:#006600">// this method is running in the working thread, which is saving the data to DB</span></i><o:p></o:p></p>
<div>
<p class="MsoNormal"><i><span style="color:#3333FF;background:white">void WorkerMethod() </span></i><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><i><span style="color:#3333FF;background:white">{</span></i><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><i><span style="color:#3333FF;background:white"> while (true)
</span><span style="color:#006600;background:white">// infinite loop is checking the queque and sending the top pair to DB</span></i><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><i><span style="color:#3333FF;background:white"> {</span></i><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><i><span style="color:#3333FF;background:white"> if (!CurrencyPairQueue.IsEmpty())</span></i><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><i><span style="color:#3333FF;background:white"> {</span></i><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><i><span style="color:#3333FF;background:white"> ServiceDatabase.SaveToDB(CurrencyPairQueue.GetTopPair());</span></i><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><i><span style="color:#3333FF;background:white"> System.Threading.Thread.Sleep(25); // sleep the thread for 25ms for lower CPU load</span></i><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><i><span style="color:#3333FF;background:white"> }</span></i><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><i><span style="color:#3333FF;background:white"> }</span></i><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><i><span style="color:#3333FF;background:white">}</span></i><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The queque is saving the data as it goes from the OnMessage event in main thread, if some currency pair come more then once, the older data is dropped. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The issue in my solution is high CPU load (cca 15% in average with 25ms sleep) of my service. I thought of several possibilities to fix this: <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">1) Get rid of the infinite loop. I thought about this, but I did not find a better way how to do it since the queque is almost never empty (any event-like solution will be very similar to this in my opinion). <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">2) Save whole queque in one insert statement. The problem is that the pairs are saving to database through storage procedure (it also process the data), so the entire process would have to be rewriten. And I am not entirely convinced it
would help. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I also tried to play with the Thread.Sleep value - the higher values is forcing too much values to drop out from the queque. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">What solution do you think will be the best? Or maybe I am all wrong and you can find another one. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Regards, <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">kwitee<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>