Hi All
I'd like to verify that my understanding of creating a stream connection to Lightstreamer is correct. I'm referring to Section 4 - The Text Mode Protocol - of the Lightstreamer Network Protocol Tutorial which is available on the IG Labs site here https://labs.ig.com/lightstreamer-downloads, and the few details given on the Guide page here https://labs.ig.com/streaming-api-guide, and also some details gathered from previous forum discussions.
To be clear about what's happening I'm using Chrome's Postman extention. Here is a preview of a post using URL Parameters: (PS. I've fiddled here with the keys but I don't believe anyone could do anything with them anyway.)
POST /lightstreamer/create_session.txt?LS_user=AE5QU&LS_password=CST-1a87ccda2d8702bb076bd50ddf1d8c0bbe0cb3f50d8b4f90ee4a3e5638c5873501112|XST-459774e733e4db98ff851780cfdeea23e0770a2563df170fb6785e0b7373ced701113&LS_adapter_set=DEFAULT HTTP/1.1
Host: demo-apd.marketdatasystems.com
Cache-Control: no-cache
And the result is:
STATUS 400 Bad Request
Next I tried using Headers instead of URL parameters which is more encouraging, but fails differently:
POST /lightstreamer/create_session.txt HTTP/1.1
Host: demo-apd.marketdatasystems.com
LS_user: AE5QU
LS_password: CST-1a87ccda2d8702bb076bd50ddf1d8c0bbe0cb3f50d8b4f90ee4a3e5638c5873501112|XST-459774e733e4db98ff851780cfdeea23e0770a2563df170fb6785e0b7373ced701113
LS_adapter_set: DEFAULT
Cache-Control: no-cache
And the result is:
STATUS 200 OK
ERROR
1
User/password check failed
I've also tried various combinations and content-types putting the parameters in the body of the request. These either give the same password check failure, or fail to return anything at all.
The same pattern of credentials works fine using the Streaming Companion and inspecting the messages using Chrome's developer tools, so at this point I confess I'm stuck.
Can anyone shed some light on what going wrong here?
Best regards
Neil
Parameters to POST should go in the body of the message. Try:
POST /lightstreamer/create_session.txt HTTP/1.1
Host: demo-apd.marketdatasystems.com
Cache-Control: no-cache
LS_user=AE5QU&LS_password=CST-1a87ccda2d8702bb076bd50ddf1d8c0bbe0cb3f50d8b4f90ee4a3e5638c5873501112|XST-459774e733e4db98ff851780cfdeea23e0770a2563df170fb6785e0b7373ced701113&LS_adapter_set=DEFAULT
Thanks for the reply APItrader. It's been a while but I wanted to follow up with the next obstacle.
I submit the post as constructed below:
url =
https://demo-apd.marketdatasystems.com/lightstreamer/create_session.txt
body =
LS_user=AE5QU&LS_password=CST-93f4b316bfc8200bdffdd6ff15610a36b663035ce487f1bd9864dfa93120eddc501111|XST-041c8133ec3071816c75bf105d2cceb4a7de54ac8a72bae79faff9fea8b2925301112&LS_adapter_set=DEFAULT
headers(1) =
name: 'Cache-Control'
value: 'no-cache'
headers(2) =
name: 'Content-Type'
value: 'application/x-www-form-urlencoded'
I'm using a well documented Matlab function urlread2 (not Matlab's own simpler urlread) but I get exactly the same behaviour if I try some other development tools, which is it just hangs and never returns either an error or a session token.
e.g. [output,extras] = urlread2(url,'post',body,headers)
I know I'm mostly right because if I mess with the password or the adapter_set values then I get sensible errors but never the expected session token.
ERROR
1
User/password check failed
- or -
ERROR
2
Requested Adapter Set not available
What am I missing?
Any help appreciated.
Neil
neilcaithness,
In case you haven't yet worked out the reason for this, what has happened is that you have opened a streaming connection to the server. The connection is being kept open by the server which is why your request appears to hang. If you are using a networking library that allows you to query the content of the stream then you can retrieve its data. The reason that using polling=true appears to work is that in this case the server closes the connection immediately and starts to accumulate the data you have subscribed to, waiting for you to "poll" it by opening a new connection to retrieve the data.
Implementing a client that can read the stream means working with a fairly low level networking library and passing the response to a separate thread that keeps querying the stream until it stops sending data. Not that difficult to do, but the lack of good documentation from IG makes it harder than it should do.
I must say I now suspect this is a problem on the IG server side. Can anyone confirm a connection using the LightStreamer plain-text protocol? i.e. not using any of the sdk's like java or .net etc.
Thanks
Neil
Hi neilcaithness
Have you tried using your clientId as the LS_user, instead of your accountId? The clientId is an integer value that comes in the response of /session
Hi belfort
Thanks for the suggestion. Yes, I'm sure I tried it before, and just tried again now. Same behaviour I'm afraid. i.e. hangs with no response.
Do you get any response if you set the body to an empty string?
If you use GET instead of POST do you get the same output as if you go to the url with your browser?
If not, your Matlab function may not be doing what you think it should do.
Have you checked the python example? http://demos.lightstreamer.com/?p=lightstreamer&t=client&f=all&lclient=p...
The interesting thing is that this example uses Text mode, but includes some parameters like LS_op2 and LS_cid that are not documented in version 5.1.2 of the Generic platform which is linked in downloads, so the example may be for a later version, or the server is using a later version
The thing is that all the client libraries except javascript and flash are using Text mode, so Text mode may not be broken, it must be the parameters.
Oh sorry forget about versions, the right python demo for 5.1.2 is this: https://github.com/Weswit/Lightstreamer-example-StockList-client-python/...
You can also check the python implementation for IG that user femto posted here: https://labs.ig.com/comment/229#comment-229
Thanks all for the suggestions. I've replicated all the behavour using Chrome's Postman extension so it's not a Matlab issue.
Interesting that the Python demo uses the LS-op2 and LS_cit parameters.
I found these documented here http://www.lightstreamer.com/docs/client_generic_base/Network%20Protocol.... Thats v.6.0.1 last updated: 04/02/2015
LS_op2=create operation description that identifies the session creation on the server
LS_cid=mgQkwtwdysogQz2BJ4Ji%20kOj2Bg magic key to be specified as is (it is
already in URL encoded form) and that identifies the protocol in use
I tried adding these two but no luck - I still get hanging no response.
Even more convinced this is a problem on the IG server side now.
Works fine for me so if it's an IG problem it is selective. I have tried with an empty body and I still get a response. If you don't, I doubt that adding parameters will solve your problem.
With an empty body, i.e. posting just this
POST /lightstreamer/create_session.txt HTTP/1.1
Host: demo-apd.marketdatasystems.com
Cache-Control: no-cache
I get
ERROR
1
User/password check failed
as I'd expect.
OK, so you are communicating okay with the server. Like I said, it works for me. The only (body) parameters that I set to log in are LS_user (which I set to my ClientId), LS_password, and LS_content_length (which is supposedly optional).
So, posting this from Chrome's Postman extension (the string below is what is shown in the preview, values edited slightly, LS_password constructed after first logging in with the ordinary API)
POST /lightstreamer/create_session.txt HTTP/1.1
Host: demo-apd.marketdatasystems.com
Cache-Control: no-cache
Content-Type: application/x-www-form-urlencoded
LS_user=100060462&LS_password=CST-c5aa1275f731c0ddaee5702729bc993aa3de585fbe25ba33753930d22c56332d01112%7CXST-f971feca465c2d336016e3d91bfa2101d194a957bed04153b144d23661b41a4901113&LS_content_length=201
I get, as before, exactly nothing. i.e Postman reports "Fetching data..." with the message "Loading..." and a cancel button.
LS_content_length is the maximum number of bytes you want to receive on the stream, not the byte count of your request, so you may want to set it higher than 201 (although, it shouldn't matter for this test). You may also want to try removing the two headers ("Cache-Control" and "Content-Type") and use a "|" instead of "%7C" in the LS-password string.
Thanks. I've set LS_content_length=1000 for testing. "%7C" is the url-encoded version of "|", but actually I do use the latter anyway. Also removed the headers as you suggest. Still nothing.
OK. I'm running out of suggestions. My last one is to try HTTP 1.0 instead of 1.1...
:-) Can't see how to force HTTP 1.0 instead of 1.1. Can it be specified in a header?
Hi all,
Hi Neil,
I experienced the same behaviour, hanging / can't fetch response.
So I tried with some LS_parameters used by the Streaming companion, and here is what's worked for me :
LS__user=A12345 // put your account ID from "accounts": [{ "accountId": "A12345", "accountName": "CFD", "preferred": true,
// accountType": "CFD" }],
LS_password=xxxxx // looks ok from your original post
LS_polling=true // same parameters as streaming companion
LS_polling_millis=0 // only if polling is true
LS_idle_millis=0 // only if polling is true
Post header
Content-type // looks ok
Content-length: // size of body (ie : LS_parameters string)
Connection: Close
Cache-Control: no-cache
I get a 'Ok' with the SessionId for rebinding connection. I'm still doing tests and I'm not sure it's the right way to use it.
Hope this will help.
Hurray, I get a response! Thanks Hyde (and all the others who've been making suggestions.)
OK
SessionId:S101fef977de41f81T4609447
ControlAddress:apd119a.marketdatasystems.com
KeepaliveMillis:0
MaxBandwidth:0.0
LOOP
Bizarrely it seems to make no difference if I use the clientId or the currentAccountId, both return with OK.
Great (although somewhat strange) that it's working. Does it also work if you set LS_polling=false (which should be the default)?
No, LS_polling=false hangs as before.
Seem to need all three:
LS_polling=true
LS_polling_millis=0
LS_idle_millis=0
Same here. I had suspected that using LS_polling = false and then sending a control request from a different console instance would have precipitated a response to the open non-polling connection, but of course if you don't receive a response you can't access the session id.
Given all the "evidence" I think that your problem may be that "urlread2" is waiting for an EOF that will never come in non-polling mode. You need to connect in some way that returns character-by-character or line-by-line without waiting for EOF. (I have no idea how to do this in Matlab.)
Yes, that does make sense, though I do get the same using Postman. Though of course that could be having the same problem.
At the point of hanging Matlab is deep in java libraries which I know little about. I'll see what I can make of it.
byteArrayOutputStream = java.io.ByteArrayOutputStream;
% This StreamCopier is unsupported and may change at any time.
isc = InterruptibleStreamCopier.getInterruptibleStreamCopier;
isc.copyStream(inputStream,byteArrayOutputStream);
inputStream.close;
byteArrayOutputStream.close;
Thank you so much for the posts above, otherwise I would not have been able to make this ;)
Now, I have one question:
- I pick up tick data from Lightstreamer every second
- After about an hour or roughly 2500-3000 data, I get a response SYNC error from LS
- My program tries to reconnect automatically (including a new session) which in first place works
- But still, Lightstreamer does not continue to work when I try to bind the session just opened.
After 5 reconnects the REST API returns a "generic security error"
- When I wait some more time, the program works ok again.
Anyone has an explanation? Am I exceeding quote (which I think not according to what I read). Is there
a limit how often you can send bind-session per hour?
Thanks for every hint!
Hello,
First of all sorry for my english.
You really did a good job coding a lightstream client for Matlab.
Can you please help me putting this code working with urlread2?
byteArrayOutputStream = java.io.ByteArrayOutputStream;
% This StreamCopier is unsupported and may change at any time.
isc = InterruptibleStreamCopier.getInterruptibleStreamCopier;
isc.copyStream(inputStream,byteArrayOutputStream);
inputStream.close;
byteArrayOutputStream.close;
Thank you!
Hi Everybody,
thanks for the great insights. Putting everything together, I was able to establish a connection via urlread2 and subscribe to LS with my account details. What I miss, now, is the way to bind the session and receive actual streaming data. I checked the manual section related with "binding" and read the following informative thread:
http://stackoverflow.com/questions/29513250/lightstreamer-client-for-matlab
but my attempts had been so far unsuccessful. I tried both a 'POST' and a 'GET' command with the following url command:
'https://apd.marketdatasystems.com/lightstreamer/bind_session.txt?LS_sess...
but matlab either stucks (with 'GET') or gives an error (when using 'POST').
May I ask for a suggestion?
Thanks a lot
Ok, eventually I was able to obtain a much expected reply to one of my data queries (e.g. DJI daily variation). However, I am now struggling in order to understand how to "stream" server response in matlab. In fact, while I am able to obtain the server reply once, I do not understand why the server is silent to iterated interrogations (ie. a FOR loop condition with bind_session.txt). It works for the first iteration and then no further data is feeded...
Well, the reading of LS manual, and, in particular, of MERGE functioning, would have saved me at least a couple of days...
thanks anyway
Hi all,
After reading the above comments I tried to create a lightstreamer session using chrome-extension POSTMAN.
As suggested I used the following parameters for a POST action:
*Host*
https://demo-aad.marketdatasystems.com/lightstreamer/create_session.txt
*Header*
Cache-Control = no-cache
Content-Type = application/x-www-form-urlencoded
*Raw Body*
LS_user=XDEHX&LS_password=CST-b76197bb91e14966e8ebdc72065a0286700cc6aa9df789028277dd2c9af7ffa401112|XST-070d9adfec96c1937bf4aac8b3f01ef6ba6180d9e3a172a659db54c9fb3bd7f201113&LS_polling=true&LS_polling_millis=0&LS_idle_millis=0
As results I received an ERROR saying: User/password check failed. Interestingly, I tried different options including using clientId instead of accountId as LS_user; adding or removing optional fields from body and header. However I still see the same error.
Can you please suggest what I am missing? I.e. do I need to request privileges in order to use lightstreamer? (so far I can successfully use the REST API). Or am I passing any incorrect values in the POST action?
Finally, I also tried using MATLAB urlread2 and I observed the following below error, any suggestion??
Error using urlread2 (line 185)
Java exception occurred:
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
at
com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(Unknown
Source)......
Correction: the *Host* is https://demo-apd.marketdatasystems.com/lightstreamer/create_session.txt
I did a little step forward, in fact I managed to get chrome-extension POSTMAN working and returning a SessionId. Here the vales I used:
*Host*
https://demo-apd.marketdatasystems.com/lightstreamer/create_session.txt
*Header*
empty, no values required
*Raw Body*
LS_cid=mgQkwtwdysogQz2BJ4Ji+kOj2Bg&LS_password=CST-facaedb756aaad619548ff6252a134d03fcf85185dd08d04d579b7c597409db001113%7CXST-462d5d4d2f5ffeffe9ee007df27c0db28424ae2cedf9169856e1bb70b216ec8e01122&LS_user=XDEHX&LS_op2=create&LS_polling=true&LS_polling_millis=0&LS_idle_millis=0
Please note that LS_cid is set to mgQkwtwdysogQz2BJ4Ji+kOj2Bg (with the + instead of space)
Regarding my Matlab implementation I am still stack with urlread2, please provide some support if possible.
Correction: the *Host* is https://demo-apd.marketdatasystems.com/lightstreamer/create_session.txt
Hi Brego81,
To see the streaming API working, you may want to look at the Streaming API Companion, https://labs.ig.com/sample-apps/streaming-companion/index.html. To see some examples for coding for the Streaming API, please check the samples we provide in a few different programming languages, https://labs.ig.com/sample-apps, namely Excel VBA, Java, Javascript and C#.
It's not clear what programming language you're using but the message board has information for how to connect using Mathlab. There appears to be some important parameters that are required to get the connection working using Mathlab. Note that Lightstreamer doesn't have a LSClient library available for Mathlab, which is why the connection requires more effort.
There is some information on https://labs.ig.com/streaming-api-guide and an example using Javascript, which illustrates some of the concepts involved.
Hope this helps!
Chris
Chris thanks for your comment.
As you pointed out the Matlab does not have a LSClient library, however neilcaithness got it working hence I thought he could give me an hand (but unfortunately I cannot contact him directly).
Having said that, I am working with Matlab on a Linux machine and I managed to run lightstreamer using a workaround since urlread2 works partially for me. As reported below in the code, the workaround involves using curl.
In addition, I decided to share my Matlab IG API implementation which is now available at https://github.com/brego81/IG_Trading_API_for_Matlab (feel free to contribute). I hope future Matlab users will benefit from it.
Finally, the below lightstreamer implementation requires urlread2 and parse_json both available from the above github link.
%%%%%========%%%%%%
clear all; clc;
X_IG_API_KEY = 'XXXXX';
User = 'XXXXX';
Password = 'XXXXXXX';
%% 1) Login
URL = 'https://demo-api.ig.com/gateway/deal/session';
header(1).name = 'Content-Type';
header(1).value ='application/json; charset=UTF-8';
header(2).name = 'Accept';
header(2).value ='application/json; charset=UTF-8';
header(3).name = 'X-IG-API-KEY';
header(3).value = X_IG_API_KEY;
header(4).name = 'version';
header(4).value ='2';
body = ['{"identifier": "' User '", "password": "' Password '", "encryptedPassword": null}'];
[output,extras] = urlread2(URL, 'POST', body, header);
[data json] = parse_json(output);
currentAccountId = data{1}.currentAccountId;
lightstreamerEndpoint = data{1}.lightstreamerEndpoint;
X_SECURITY_TOKEN = extras.firstHeaders.X_SECURITY_TOKEN;
CST = extras.firstHeaders.CST;
%% 2) Create Lightstreame session using create_session.txt
body_LS_curl = ['LS_cid=mgQkwtwdysogQz2BJ4Ji+kOj2Bg&'...
'LS_password=CST-' CST '%7CXST-' X_SECURITY_TOKEN '&'...
'LS_user=' currentAccountId '&'...
'LS_op2=create&'...
'LS_polling=true&LS_polling_millis=0&LS_idle_millis=0&LS_adapter_set=DEFAULT'];
body_LS_curl = [' -d ''', body_LS_curl ,''''];
post = [' -X POST ' lightstreamerEndpoint '/lightstreamer/create_session.txt'];
[status, cmdout] = unix(['LD_LIBRARY_PATH=""; curl -s ', '', post, body_LS_curl]);
if strcmp(cmdout(1:2), 'OK')
new_line = strfind(cmdout, char(10));
f = strfind(cmdout, 'SessionId:');
SessionId = cmdout(f+10:new_line(2)-2);
f = strfind(cmdout, 'ControlAddress:');
lightstreamerEndpoint = ['https://' cmdout(f+15:new_line(3)-2)];
end
%% 3) Request data using control.txt
symbols = {'IX.D.MIB.IFD.IP' 'IX.D.CAC.IFD.IP' 'IX.D.DAX.IFD.IP' 'IX.D.IBEX.IFD.IP'};
symbols_text = ['L1%3A' symbols{1}];
for s = 2 : size(symbols,2)
symbols_text = [ symbols_text '+L1%3A' symbols{s}];
end
body_LS_curl_1 = ['LS_mode=MERGE&'...
'LS_session=' SessionId '&'...
'LS_Table=1&LS_id=' symbols_text '&' ...
'LS_op=add&LS_schema=UPDATE_TIME+BID+OFFER+MARKET_STATE' '&' ...
'LS_polling=true&LS_polling_millis=0&LS_idle_millis=0'];
body_LS_curl_1 = [' -d ''', body_LS_curl_1 ,''''];
post = [' -X POST ' lightstreamerEndpoint '/lightstreamer/control.txt'];
[status,cmdout] = unix(['LD_LIBRARY_PATH=""; curl -s ', post, body_LS_curl_1]);
%% 4) Stream values using bind_session.txt
body_LS_curl_2 = ['LS_session=' SessionId '&LS_polling=true&LS_polling_millis=0&LS_idle_millis=0'];
body_LS_curl_2 = [' -d ''', body_LS_curl_2 ,''''];
post = [' -X POST ' lightstreamerEndpoint '/lightstreamer/bind_session.txt'];
for t =1:10
[status, cmdout] = unix(['LD_LIBRARY_PATH=""; curl -s ', post, body_LS_curl_2]);
disp(cmdout(126:end-10));
% 1,1 -> IX.D.MIB.IFD.IP
% 1,2 -> IX.D.CAC.IFD.IP
% 1,3 -> IX.D.DAX.IFD.IP
% 1,4 -> IX.D.IBEX.IFD.IP
pause(0.3);
end
Here are my comments on the topic:
POST /lightstreamer/create_session.txt HTTP/1.1
Content-Length: 195
Content-Type: text/plain
LS_password=CST-e4c00ba9a68872740ed000d13f353894ef5809d249b0f207296f5dbd02d9f98001112|XST-9f2fc3953f8202447868a893ff9ec15eb6f7a9ecd5278dff5df8b4a35a2ff79a01111&LS_polling=true&LS_polling_millis=0
So: I took as content type "text/plain", didn't set the content length (it seems my http-library casablanca sets it automatically) and then just added the three queries LS_password, LS_polling and LS_polling_millis as above. It seems to suffice -- which also explains the confusion whether to pass the accountId or the userId in LS_user -- it simply seems to get neglected by the API.
Here are my comments on the topic:
POST /lightstreamer/create_session.txt HTTP/1.1
Content-Length: 195
Content-Type: text/plain
LS_password=CST-e4c00ba9a68872740ed000d13f353894ef5809d249b0f207296f5dbd02d9f98001112|XST-9f2fc3953f8202447868a893ff9ec15eb6f7a9ecd5278dff5df8b4a35a2ff79a01111&LS_polling=true&LS_polling_millis=0
So: I took as content type "text/plain", didn't set the content length (it seems my http-library casablanca sets it automatically) and then just added the three queries LS_password, LS_polling and LS_polling_millis as above. It seems to suffice -- which also explains the confusion whether to pass the accountId or the userId in LS_user -- it simply seems to get neglected by the API.
Just continuing, as I hit save too early:
OK
SessionId:Sa9c6c31d19515569T1315166
ControlAddress:apd120a.marketdatasystems.com
KeepaliveMillis:0
MaxBandwidth:0.0
LOOP
That's it for /lightstreamer/create_session.txt, I guess. So far so good.
POST /lightstreamer/control.txt HTTP/1.1
Content-Length: 130
Content-Type: text/plain
LS_session=Sa9c6c31d19515569T1315166&LS_op=add&LS_id=L1:CS.D.EURUSD.CFD.IP&LS_schema=BID&LS_mode=MERGE&LS_table=1
Again, I haven't explicitly set the Content-Length. This shall do a subscription to the MARKET-item of the EUR/USD (Epic: CS.D.EURUSD.CFD.IP), and particularly the bid-price -- don't ask me why it's "L1" instead of "MARKET", but it seems to work as the reponse is a plain
OK
That's it for the solutions, next come the questions.
Hi davidhigh,
Thanks for your comments, these are indeed good points. This will need addressing, and we'll look at this internally.
Thanks,
Chris
Hi Chris,
Did you ever get round to writing this guide? It's really difficult to work out what you're supposed to do with the current material - sort of like feeling around in the dark.
James