{{Quickfixn}} Reconnecting problem

[Thomas Tomiczek]

What I do is:


·         Event driven queue spawning a worker thread when data arrives and non is spawned.

·         Using efficient insert mechanisms. Sorry to say, but I am managing 75000 new data activity entries per second from the write loop, and that is on lower end hardware, limited by the database server IO. Not sure what you use as Database, but for SqlServer do NOT generate INSERT statements, use SqlBulkCopy, in large increments (I think I use 50.000 rows per insert now).

There should not be processing of any incoming data just for recording, and definitely NOT in the database. – sorry. But that is definitely not at all a QuickFix question anymore.

Just as note – I have no problem most of the time keeping up with the complete CME group stream (which is around 200.000 symbols, though most very rarely active). With most capacity left.

Regards

Thomas

From: quickfixn-bounces at lists.quickfixn.com [mailto:quickfixn-bounces at lists.quickfixn.com] On Behalf Of Ondra Kvet
Sent: 17 July 2012 16:08
To: Mailing list for QuickFIX/n
Subject: Re: {{Quickfixn}} Reconnecting problem

Hi everyone,
I have found another problem relating to this. I am simply asking the provider for market data and saving them to database. I implemented Grant's solution in following way:

// this method is running in the working thread, which is saving the data to DB
void WorkerMethod()
{
  while (true) // infinite loop is checking the queque and sending the top pair to DB
  {
    if (!CurrencyPairQueue.IsEmpty())
    {
      ServiceDatabase.SaveToDB(CurrencyPairQueue.GetTopPair());
      System.Threading.Thread.Sleep(25); // sleep the thread for 25ms for lower CPU load
    }
  }
}

The queque is saving the data as it goes from the OnMessage event in main thread, if some currency pair come more then once, the older data is dropped.

The issue in my solution is high CPU load (cca 15% in average with 25ms sleep) of my service. I thought of several possibilities to fix this:

1) Get rid of the infinite loop. I thought about this, but I did not find a better way how to do it since the queque is almost never empty (any event-like solution will be very similar to this in my opinion).
2) Save whole queque in one insert statement. The problem is that the pairs are saving to database through storage procedure (it also process the data), so the entire process would have to be rewriten. And I am not entirely convinced it would help.

I also tried to play with the Thread.Sleep value - the higher values is forcing too much values to drop out from the queque.

What solution do you think will be the best? Or maybe I am all wrong and you can find another one.

Regards,
kwitee

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.quickfixn.com/pipermail/quickfixn-quickfixn.com/attachments/20120717/12a57211/attachment-0002.htm>