{{Quickfixn}} Latency in log writing

[オイゲン デュック]

In case you are using FileLog:

Both the message log and the event log have their StreamWriter.AutoFlush set to "true". The source can be found at https://github.com/connamara/quickfixn/blob/master/QuickFIXn/FileLog.cs#L41

This may hurt performance, because after every single write to the logs, FileStream.Flush(flushToDisk: false) will be called internally. (Which however should not be as bad as flushToDisk = true)
What you could do is as a test: Clone QuickFixN's FileLog (and you also have to clone FileLogFactory to be able to use it) into your own project and turn off AutoFlush there. If that does in fact remove the latency spikes you are seeing, then you can have a look at how to address the frequent flushing. There are a couple of ways to do flushing in a less performance impacting, yet low-latency way.

Eugen

On 2019/05/15 4:03, Grant Birchmeier wrote:
I won't pretend I have special knowledge here; I've never looked at logging performance before in QF/n.  It could be Windows file IO, or maybe there's something unintentionally silly happening in QF's internals.

To investigate, you could write your own LogFactory implementation and integrate some profiling into it.  The interface is pretty simple.

On Tue, May 14, 2019 at 1:55 PM xmoon 2000 <xmoon2000 at googlemail.com<mailto:xmoon2000 at googlemail.com>> wrote:
Grant,

In your opinion, should QuickFix be able to write to its own log
without causing latencies, if as in my code snippet, I do nothing
except return from                             FromApp WITHOUT calling
Crack.

For approx 20m messages per day?

On Tue, 14 May 2019 at 15:42, Grant Birchmeier
<gbirchmeier at connamara.com<mailto:gbirchmeier at connamara.com>> wrote:
>
> Is this latency causing performance problems other than the logging delay?  Is the delay constant, or does it only occur when there's a huge influx of messages?  (Would it get worse than 500ms if the high-traffic continued?)
>
>
> On Tue, May 14, 2019 at 9:14 AM xmoon 2000 <xmoon2000 at googlemail.com<mailto:xmoon2000 at googlemail.com>> wrote:
>>
>> Hi,
>>
>> I get about 20 million records via QuickFix .NET in c# per day.
>>
>> I have done a pcapture and can see that the tag52 and WireShark times
>> are close - so latency is not from transmission.
>>
>> In my app I have stripped everything out (see FromApp below).So that I
>> do nothing with the data - I just allow QuickFix engine to write fix
>> messages to log on my SSD.
>>
>> BUT I can see latencies of 500ms between QuickFix log time and tag52 time.
>>
>> What might be causing this? Is writing to SSD so slow? Or is it
>> because several other apps are reading this log file?
>>
>>         public void FromApp(QuickFix.Message msg, SessionID sessionID)
>>         {
>>             return;
>>
>>             try
>>             {
>>                 Crack(msg, sessionID);
>>             }
>>             catch (Exception ex)
>>             {
>>                 Console.WriteLine("==Cracker exception==");
>>                 Console.WriteLine(ex.ToString());
>>                 Console.WriteLine(ex.StackTrace);
>>             }
>>         }
>> _______________________________________________
>> Quickfixn mailing list
>> Quickfixn at lists.quickfixn.com<mailto:Quickfixn at lists.quickfixn.com>
>> http://lists.quickfixn.com/listinfo.cgi/quickfixn-quickfixn.com
>
>
>
> --
> Grant Birchmeier
> Connamara Systems, LLC
> Made-To-Measure Trading Solutions.
> Exactly what you need. No more. No less.
> http://connamara.com
> _______________________________________________
> Quickfixn mailing list
> Quickfixn at lists.quickfixn.com<mailto:Quickfixn at lists.quickfixn.com>
> http://lists.quickfixn.com/listinfo.cgi/quickfixn-quickfixn.com
_______________________________________________
Quickfixn mailing list
Quickfixn at lists.quickfixn.com<mailto:Quickfixn at lists.quickfixn.com>
http://lists.quickfixn.com/listinfo.cgi/quickfixn-quickfixn.com


--
Grant Birchmeier
Connamara Systems, LLC
Made-To-Measure Trading Solutions.
Exactly what you need. No more. No less.
http://connamara.com



_______________________________________________
Quickfixn mailing list
Quickfixn at lists.quickfixn.com<mailto:Quickfixn at lists.quickfixn.com>
http://lists.quickfixn.com/listinfo.cgi/quickfixn-quickfixn.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.quickfixn.com/pipermail/quickfixn-quickfixn.com/attachments/20190514/9b3b02a0/attachment.htm>