{{Quickfixn}} High CPU Usage

[Mike Gatny]

Steve,

When you create your Initiator, you tell it which MessageStoreFactory and
LogFactory to use (the latter is optional and defaults to NullLog, which
logs nothing).  If you use MemoryStoreFactory, you will prevent storing
session info (e.g. seqnums) to file, etc.:

QuickFix.Transport.SocketInitiator initiator = new
QuickFix.Transport.SocketInitiator(application, new
 QuickFix.MemoryStoreFactory(), settings);

However, FileStore would have been the more likely culprit -- if you
already aren't using it, I don't think MemoryStore is going to help much.

There is surely some low-hanging fruit in terms of optimizing qf/n message
parsing, etc.  But if you can break out subs onto more Sessions/instances,
it feels like that's the short-term solution.



--
Mike Gatny
Connamara Systems, LLC

On Fri, Mar 2, 2018 at 12:31 PM, Steven Leonard <stois7644 at gmail.com> wrote:

> Hi Mike - thanks for the reply.
>
> I haven't implemented any logging within QuickFix, I'm handling it myself
> within the FromApp but its pretty efficient and as you said I have already
> tried removing it and still couldn't keep up.
>
> Can you explain the nullLog + MemoryStore combination you suggest? I
> assume the nullLog is already in place since I don't define a FileLogPath? I
> see no settings relating to Memory...
>
> On Fri, Mar 2, 2018 at 4:17 PM, Mike Gatny <mgatny at connamara.com> wrote:
>
>> Steve,
>>
>> The usual cause is doing too much processing in the FromApp/OnMessage
>> callback, which happens on QF/n's message processing thread, and therefore
>> blocks it from progressing until you return from the callback.
>>
>> If that is not the case (you mentioned you removed all processing to see
>> what would happen, which was a good idea), then splitting your
>> subscriptions even further onto separate instances of the app is indeed a
>> proven strategy.
>>
>> One other thing to look at: which Log and Store implementations are you
>> using?  It might be interesting to try the combination of NullLog +
>> MemoryStore.
>>
>>
>> --
>> Mike Gatny
>> Connamara Systems, LLC
>>
>> On Fri, Mar 2, 2018 at 9:47 AM, Steven Leonard <stois7644 at gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I am using the Quickfix .NET dll v1.7 to export raw
>>> FIX MarketDataSnapshotFullRefresh (35=W) messages to a flat file.
>>>
>>> It seems I have been hitting an upper limit on the message rate Quickfix
>>> can handle. It maxes out each 3.10Ghz Xeon core when I get towards 500
>>> messages per second. The messages contain up to 25 Bids and 25 offers
>>> (Depth of 50).
>>> The app then cannot keep up with the messages being sent - I keep
>>> getting disconnected by the sender as the message buffer their side has
>>> reached the maximum depth they allow.
>>>
>>> Has anyone else hit this high CPU usage issue? I have tried removing any
>>> processing my side to the point where I don't even crack the messages and
>>> have even tried a version which doesn't do anything but consume the
>>> messages (no logging/output) without luck.
>>>
>>> Any suggestions to improve performance/efficiency are welcome.
>>> Else I can only split the symbols out into more instances of the app and
>>> use more CPU cores - I'm already running three.
>>>
>>> Thanks,
>>>
>>> Steve
>>>
>>> _______________________________________________
>>> Quickfixn mailing list
>>> Quickfixn at lists.quickfixn.com
>>> http://lists.quickfixn.com/listinfo.cgi/quickfixn-quickfixn.com
>>>
>>>
>>
>> _______________________________________________
>> Quickfixn mailing list
>> Quickfixn at lists.quickfixn.com
>> http://lists.quickfixn.com/listinfo.cgi/quickfixn-quickfixn.com
>>
>>
>
> _______________________________________________
> Quickfixn mailing list
> Quickfixn at lists.quickfixn.com
> http://lists.quickfixn.com/listinfo.cgi/quickfixn-quickfixn.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.quickfixn.com/pipermail/quickfixn-quickfixn.com/attachments/20180302/ba9d2824/attachment.htm>