How to fetch dynamic JSON file content?

[LCF-AT]

Hi guys,

so I would like to fetch entire datas from a dynamic JSON file but I have some problems to do it to update cookies without to work in browser directly.So some sites using such auto load functions if you do scroll up / down to show the actually content and or you have to click a button to let show more content if present etc you know what I mean right.Now I tried to find out how it works (checking traffic in webconsole) and how I could code something to do same without visiting the site but here I have the problem and dont know how to get entire content.Problem is it works with diffrent cookies which get changed like session and other id cookie entrys which I should use for next request to get next JSON content till last etc.Seems to be not so easy as I thought before also if I dont know how to get actually cookie values withtout to work in browser / scrolling etc.Now I would like to ask for any ideas how I could handle that or whether there are also other ways to get entire content.

greetz

[evlncrn8]

InternetSetCookie https://msdn.microsoft.com/en-us/library/windows/desktop/aa385107(v=vs.85).aspx

InternetGetCookie https://msdn.microsoft.com/en-us/library/windows/desktop/aa384710(v=vs.85).aspx

assuming you're using wininet 

[secursig]

You'd have to write code that captures the returned server headers from each request, starting from the very first one where any session variables, tokens, or other data are assigned. With each subsequent request, you also need to check to see what is changed in the cookie (by examining the server's response headers each time) and track those data as a "global" variable when sending the next request...since the server is going to be expecting them.

Of course this is easier when the browser does it for you. If it's too complicated, I'd probably just write some javascript function and load it in my browser that can interact with the code running on that page....for example, make a function that simulates the scrolling and thus automatically sends all the requests from the browser. you should use another function to dump the response from the ajax queries, hook the callbacks ( javascript uses a lot of these )...or just route your browser through a proxy server ( with root CA installed if the site is SSL ), and then from a utility such as mitmproxy you can dump all the ajax requests to disk easily and then write a simple script to assemble them back to binary or whatever format is easiest. (mitmproxy can have plugins you can write)

 

that's what I'd do.

[LCF-AT]

Hi again,

so I cant capture something what only gets loaded by scrolling in browser.So how should I do it without browser / scrolling?So if I scroll then it will request new link to JSON file which at the end has a value X I can also find inside the JSON file as page ID but before I cant find nothing in webconsole how is has get that value X.Also one problem.If I do scroll forward then same JSON link with same X value gets loaded another time (new content inside) and the request was doing with all same cookie values so they dont change and this process is going on so long till last page.This process I cant reproduce so I tried it but my cookie value are changing always and then I only get results like from first request and come not forward to get the other datas till last.So I dont get it.

PS: I am using WInsock in that case.

greetz