Order of data in buffets

[giovannetti87]

Dear all,

Assuming a circular buffer of size 20, obtaining some market data, call it Buf. I would like to confirm the chronological structure of the data. Is it always the case that Buf(end) contains more recent (or simply, no older ) data than Buf(1) ?
thanks,

[Despair2000]

If you use reqMarketData you can be sure that they come in chronological order and these are FIFO buffers. So the first event is always the most recent and last the oldest.
If this is also true when requesting historical data I'm not sure. Abel sorts the events in his example so he maybe has experienced that they were not in the right order.

[giovannetti87]

Thanks Despair!

[softwarespartan]

yes, these are FIFO buffers. Again, keep in mind that you don't have to use buffers, per se. IB4m hands off the events to MATLAB in the exact same order as they are provided from TWS. There are cases where you might want to be defensive. For example, historical data request is just good practice to sort the data (i.e. it can't hurt, but maybe not required) If processing data in real-time, then might just keep track of the most-recent datetime. If any event has a datetime less than your latest datetime then could reject that datapoint. Be sure to keep in mind that there are two times here. one is the event time provided by IB servers then the second time is the time associated by TWS for when the event is received and processed (does that make sense?). This is very clear for historical data, but can sometimes be hard to differentiate for real-time market data since those times are very close. Therefore, event.data.datetime is the datetime associated with the data, and event.datetime is the datetime associated with the event itself.

…

On Fri, Sep 3, 2021 at 5:50 AM giovannetti87 ***@***.***> wrote: Thanks Despair! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#139 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABC2VVFEKEBYWHCDLRXQWELUACK5JANCNFSM5DGXXOHA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[giovannetti87]

Hi @softwarespartan , thanks for the reply and the generous work on this great project. I spent some time reading your previous replies on the topic to some user who had a similar situation and I will likely remove buffers from my current script as it is entirely based upon real time data feed. Hopefully, this will also allow me to solve the problem of matlab freezing upon script termination (ctrl+C) when a large number of buffers is deployed. This behaviour is still puzzling me.

I would like to reiterate my gratitude toward you two guys for your support. I literally went from zero into having an almost-fully operative script with data collection and order making capabilities.

[Despair2000]

The gratitude that I share deserves only Abel. I'm just a user like yourself.

How many is "a large number of buffers"? I've been using 80 and more simultaneously with out problems.

[giovannetti87]

@Despair2000 Hi Despair, unfortunately I didn't see your message earlier. It appears that my data flow gets increasingly clogged when I cross 30-40 streams during US time trading session. With earlier or later times it is usually fine and I can work to saturation. I am trying to track down the problem, which it might be related to java

[softwarespartan]

Be sure to increase your JVM heap size and memory limits at Matlab launch

…

Sent from my iPhone

On Sep 30, 2021, at 10:57 PM, giovannetti87 ***@***.***> wrote:  @Despair2000 Hi Despair, unfortunately I didn't see your message earlier. It appears that my data flow gets increasingly clogged when I cross 30-40 streams during US time trading session. With earlier or later times it is usually fine and I can work to saturation. I am trying to track down the problem, which it might be related to java — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[giovannetti87]

Hi @softwarespartan , sounds great, I'll try right now. Thanks!

[giovannetti87]

Hi @softwarespartan Abel, unfortunately it seems the clogging is still happening. When I try to terminate matlab with ctrl+c, this is what matlab prints. Sometimes I am able to get the command line back to normal (after long waiting) most of times, not.

Thank you!

Operation terminated by user during TWS.processNotification (line 7) com.mathworks.jmi.MatlabException: InterruptException(ctrl-c) at com.mathworks.jmi.NativeMatlab.SendMatlabMessage(Native Method) at com.mathworks.jmi.NativeMatlab.sendMatlabMessage(NativeMatlab.java:273) at com.mathworks.jmi.MatlabLooper.sendMatlabMessage(MatlabLooper.java:120) at com.mathworks.jmi.Matlab.mtFevalConsoleOutput(Matlab.java:1835) at com.proxy.ProxyBuilder$Handler$2.run(ProxyBuilder.java:125)

[Despair2000]

For me it can also take some time for Matlab to return to the command line after I issue CTRL-C but we are talking about a few seconds.
What kind of machine are you on?

[giovannetti87]

Hi @Despair2000, I'm using an aero 15 with 16gb ram and i7, usually it doesn't trouble me too much?

[giovannetti87]

A screenshot of what I see. You can see a bunch of errors every time I try to terminate execution. I have to press CTRL+C several times before the script accepts the halt, and nonetheless the command window is never returned.

[Despair2000]

I also get several of these errors before the scripts exits and it happened to me also that a script did not return to the command window but every time this happened it was usually something with my code.
Usually I hit several times CTRL-C to break a script then I wait a moment for the command line to return. If it doesn't I restart matlab.
You don't have any fifo-buffers in a cell array? I ask because this caused really weird errors for me.

[giovannetti87]

Glad to hear that to a lesser extent, you're experiencing the same phenomenon. In my case, matlab is still workable if I keep the flow to 20-40 contemporaneous streams. Anything north this, forces a reboot. I do have several fifo-buffers in struct. variables, but I don't think I have any in cell arrays.

[softwarespartan]

@giovannetti87 can you pull the latest IB4m update from GitHub. There was a bug in the previous JAR file that has been patched in the recent update. This update will resolve the issue with proxy builder