Downloader tools - Which is a good one?

[LCF-AT]

Hi guys,

today I have a new question and need some infos to find something useful what I could use.

Maybe you know the problem,you do surf on internet and you find any file / video on any sources you wanna download.I notice that there are some or more sources / hosts who try to deny a download with full speed and you got just some KB/s what really sucks and if you know me then you know I don't like limits.In the past I have seen that problem too already and tried some different donwloader tools who have a feature called "accelerator" and in some cases it really works and I could download files with 100 % speed of my internet connection instead of some KB/s like with a browser but unfortunately isn't work anymore (tool/s outdated or Hosts getting smarter or what ever etc).All in all I don't undertsand this speed limits and why some sources do this so bad.I found also some video hosts where its almost not possible to watch a video on the site itself.30 seconds loading / one second watching and again and again so on etc this is totally ridiculous.I think you know that problems too more or less.Now I am looking again for any solution / tool/s what can handle such issues to download files fast and comfortable with almost my internet speed I also have.Does anyone of you know some good tools or AddOns etc who can do this which can download faster as browser itself?

Example: There is a video host called streamcloud.On testing I found out that the download first starts with full speed (few seconds) of my connection but very quick the speed getting slower and slower till 0.132 MB/s (132 KB/s) and to download a file with a size of 50.7 MB has taken a time of 6.36 minutes.Only for 50 MB!?!Just a example.

Maybe you know some magic what could help.

PS: Of course I am just taling about free sources / Hosts.

greetz

[NOP]

Download throttling is usually done by the server per connection

Accelerators work by making multiple connections and then piecing the download bits together once they are all downloaded but not all servers support this

I use IDM (Internet Download Manager)

LINK

[Teddy Rogers]

Perhaps the Firefox addon DownloadThemAll?

Ted.

[li0nsar3c00l]

I personally use jDownloader

[kao]

JDownloader used to be good when depositfiles, megaupload and rapidshare were all the rage. Now I don't see a point in using it - anything I want are either on torrents or on really fast servers.

To grab streaming videos: http://rg3.github.io/youtube-dl/download.html (which claims to support streamcloud, among other servers)

 

[Alzri2]

You can use cbox websites (most of the times, I use cboxforeveralone) to download files from stupid file sharing services like uploaded.

 

P.S: don't forget to use IDM in case downloading stops so you can replace the downloading link and resume

Edited April 25, 2016 by Alzri2

[LCF-AT]

Hi guys,

thanks for your answers so far.I think the accelerator DL tools are not really working.Just tried the DownloadThemAll AddOn now with streamcloud but its same slow as the download without tools. Maybe it makes no sense anymore to use any tool like this if now almost all hosts using any speed limits also if they are just free hosts.

Thanks again.

[kao]

1 hour ago, LCF-AT said:

Just tried the DownloadThemAll AddOn now with streamcloud but its same slow as the download without tools.

Are you sure it's not a problem with your ISP (eg. throttling big downloads) or your computer (eg. virus scanner trying to scan files being downloaded)?

I just tried streamcloud (random link found via google: http://streamcloud.eu/rdnnm0pmvuxx/collection-tickletime.avi.html) and it's really fast. HTML5 player is slightly faster than Flash but in both of them playback and seeking works without any lag whatsoever.

[LCF-AT]

Hi kao,

hmmm I don't use Virus scanner etc.I tried your link too but get same as I post before.First few seconds I got full download speed and then its getting slower and slower.Not happens if I download any other files from other sources like any apps etc there I get also full speed.So what do you get?Does your DL speed keep same constant or does it also getting slower till very slow?

greetz

[Techlord]

3 hours ago, LCF-AT said:

Hi kao,

hmmm I don't use Virus scanner etc.I tried your link too but get same as I post before.First few seconds I got full download speed and then its getting slower and slower.Not happens if I download any other files from other sources like any apps etc there I get also full speed.So what do you get?Does your DL speed keep same constant or does it also getting slower till very slow?

greetz

Hi LCF-AT,

Have been reading and following a lot of your posts and your work, especially in the areas of reversing WinLicense, and VMProtect for at least the past 2-3 years.

Marvellous work I must say ! Kudos to you. Even though its been 3 years, your scripts to reverse VMP and Winlicense still remain functional !

With talents like you (and others around), who are a lot more skilled than me, I do not find much opportunity to post in this forum a lot. However, I still read up the posts here regularly.

Now that I feel that you have the same problem that I had experienced with various ISPs earlier on, though I'd just contribute my two cents

Regarding your downloading issue :

I've had the same problem with many ISPs, in many countries, where I used to travel mainly for work purposes. The downloads initially start at the full speed of the internet bandwidth that I paid for, but slowly, they trickle down to a very small speed.

I have noticed that the ISPs achieve it in many ways : sometimes, they simply are looking for KEYWORDS in your url (keywords like rapidshare, uploaded, rapidgator. rar, MP4 etc) and then, then they configure their server to limit our download speeds.

Other MAIN way that that I have seen many ISPs achieve it is by simply only allowing a FIXED MB or KB from a particular URL and then simply throttling the speed, regardless of the content downloaded. For example, they allow only 5 MB from any site at full speed, then the remainder is only allowed at , say, 50 Kbps or something. Their idea being, that HTML browsing of webpages should be very fast but that larger DOWNLOADS should be slower, thereby keeping the networks of the ISPs less congested.This was very frustrating for me.

Hence the solution that I found, that would work in nearly ALL the cases (at least for me):

Use a VPN.

Some very specific settings etc that I want to clarify and stress on , that are crucial for success :

Here, stealth of content that we are downloading is not what we are after (for example, we are not trying to hide illegal torrent activity) - so we can use the FASTER VPN option (though not very secure) --->  PPTP (preferably with encryption). This would mean lesser loss of internet speed that would be inevitable, when we use a VPN.

Many VPN providers let you have a trial period to check if it works for you and if it doesn't work for you, will refund your money.

So I will keep this very short and just say what worked for me. The steps that I used :

1. Subscribed to StrongVPN (provider).

2. Chose a PPTP server nearest  to me (for example, if in California, then choose San Francisco server or LA Server).

3, Configure PPTP with encryption (NOT SSTP or other protocols as it's causing LOSS of speed when downloading and hence defeats the very purpose of us using the VPN). Do NOT use any app provided by the VPN provider but natively configure the VPN using Windows' native Network and Sharing Center settings.

4. Check using WhatisMyIP.com to ensure that the REAL IP of yours is masked. Also try checking with speedtest.net to see if you're getting a good speed. Otherwise choose another VPN server which is faster.

5. Then retry with your download (that was earlier throttled). You should be getting a higher speed.

6. Please note that the VPN Providers are alll not created equal and when I tried with a few other VPN providers, they simply did not work for me... I wanted to change my VPN provider as I felt that the StrongVPN was too expensive, but with the otehrs, the download speeds suffered.

By the way, I do not get any incentives or rewards from StrongVPN for recommending them

This solved the problem for me. Hope it does for you too.

Note : PPTP is less secure (even with encryption) than SSTP and OpenVPN Protocols but in my experience and for OUR purposes of defeating the throttle limits set by the ISPs, this works VERY WELL....

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

[LCF-AT]

Hello Techlord,

thanks for the flowers so far. Nice to see that you like to follow some of my stuff.

About the speed problems.So I am really not sure whether the problem is to find on my ISP (provider).I did check already many sites to download something and to check the DL speed to see any differences.On many hosts I get also the full download speed also on all hosts where you have to pay for so there I got never any DL speed trouble.The only problem I get are to download from any free sources like streamcloud for example or other video hosts which are more worst than SC.I also have VPN using it rarly but I do remember before a longer time I also had DL trouble on T4Y (still getting no full speed on our forum) and used VPN and the DL was much faster.No idea why.Anyway,so made a short video with my DL tool where you can see the problems with SC for example.DL starts with full speed till 6 MB and then its getting worst.All in all this sucks and makes not much sense to upgrade to a higher connection if you get always limited on such hosts if really the host itself is the problem maker.

greetz

SC Download Test.rar

[Techlord]

Hi LCF-AT,

I have gone through the files in your attached rar file from your post above and have done a few confirmatory tests of my own.

I have a bit of bad news for you I'm afraid :

1. Firstly, the server (streamcloud.eu) is NOT accessible from outside of Europe and DEFINTELY its not accessible from American IP addresses and throws a 404 Error if attempted from US addresses. Tried with the link in your file (http://streamcloud.eu/rdnnm0pmvuxx/collection-tickletime.avi.html) but throws a 404 Error as described above. Checking on this site (http://downuptime.net/) by clicking on "Check Now" under the title "Check downuptime.net availability from worldwide locations" on that site, after entering the URL to be checked for as "streamcloud.eu" shows that the site is shown as "DOWN" or 404 error in other words, from locations outside of the EU region.

2. When I used a German VPN, the link opens as usual as if nothing's the matter, and the video starts to play without issues.

3. When I extracted the video's URL to check for download speed, its different from the one you gave in your txt file (of course...) and for me it was " http://stor23.streamcloud.eu:8080/6dv75xk3eooax3ptx3zynr7mvoeux2aa7pm7lrqzrham75df6emvm4uugm/video.mp4 " . When I attempt with the url that you gave, it says "Wrong IP".

4. The bad news is that the throttling of the speed is happening at the SERVER-SIDE at streamcloud.eu, and hence unfortunately, there's nothing that we can do   . For me, the download speed was constant at 250 Kbps though my actual bandwidth even with VPN, as tested by speedtest.net and other methods is closer to 30 Mbps... The fact that the server is using a proxy to control and otherwise limit our download speed is evident as the download links show a ":8080" after the stor23.streamcloud.eu . I would say that in all probability, they're using a SQUID proxy server for that purpose, though it hardly matters for us what exact thing they use.

5. It is POSSIBLE that if we are paying customers, rather than FREE ones, that they MAY remove this limitation. This is just my speculation but I am not sure...

Sorry to say therefore, that for this particular site, since the limitation is server-side, we cannot do anything ...

Greetz

 

 

[SameerRaj]

use idm...or ninja downloader 

[kao]

@Techlord: great analysis!  If encouraged me to look at it again, so here are few notes from my side:

1) Streamcloud seems not to throttle first 5.5-6 MB of download. For those I get 400-800kb/sec+ speeds:

F:\>curl.exe "http://stor23.streamcloud.eu:8080/6dv75xk3eooax3ptx3zynr7mvoeux2aa7pm7lrqzrh6465lrd7cxy6nng4/video.mp4"
-H "Pragma: no-cache" -H "DNT: 1" -H "Accept-Encoding: identity;q=1, *;q=0" -H "Accept-Language: en-US,en;q=0.8" -H "User-Agent:
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2414.0 Safari/537.36" -H "Accept: */*" -H "Cache-Control: no
-cache" -H "Referer: http://streamcloud.eu/rdnnm0pmvuxx/collection-tickletime.avi.html" -H "Cookie: lang=english; playermode=html5" -H "Connection:
 keep-alive" --compressed -o 1.mp4
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  1  260M    1 5213k    0     0   480k      0  0:09:15  0:00:10  0:09:05  701k

2) After the magic barrier is reached, download speeds suffer a lot:

F:\>curl.exe "http://stor23.streamcloud.eu:8080/6dv75xk3eooax3ptx3zynr7mvoeux2aa7pm7lrqzrh6465lrd7cxy6nng4/video.mp4"
-H "Pragma: no-cache" -H "DNT: 1" -H "Accept-Encoding: identity;q=1, *;q=0" -H "Accept-Language: en-US,en;q=0.8" -H "User-Agent:
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2414.0 Safari/537.36" -H "Accept: */*" -H "Cache-Control: no
-cache" -H "Referer: http://streamcloud.eu/rdnnm0pmvuxx/collection-tickletime.avi.html" -H "Cookie: lang=english; playermode=html5" -H "Connection:
 keep-alive" --compressed -o 1.mp4
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  2  260M    2 5956k    0     0   335k      0  0:13:15  0:00:17  0:12:58 93249

3) But you can always close the connection and open a new one indicating "Range: bytes=xxxx". You get a full speed again:

F:\>curl.exe "http://stor23.streamcloud.eu:8080/6dv75xk3eooax3ptx3zynr7mvoeux2aa7pm7lrqzrh6465lrd7cxy6nng4/video.mp4"
-H "Pragma: no-cache" -H "DNT: 1" -H "Accept-Encoding: identity;q=1, *;q=0" -H "Accept-Language: en-US,en;q=0.8" -H "User-Agent:
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2414.0 Safari/537.36" -H "Accept: */*" -H "Cache-Control: no
-cache" -H "Referer: http://streamcloud.eu/rdnnm0pmvuxx/collection-tickletime.avi.html" -H "Cookie: lang=english; playermode=html5" -H "Connection:
 keep-alive" -H "Range: bytes=6000000-" --compressed -o 2.mp4
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  2  254M    2 5538k    0     0   457k      0  0:09:30  0:00:12  0:09:18  623k

That's why seeking in the video file works so well.

 

So, the easy way around streamcloud's limitations would be downloading 5-6MBs, closing a connection and opening a new one.

Edited April 29, 2016 by kao
hide some private info...

[LCF-AT]

Hi again,

thank your checking the site issues Techlord. So I thought already that is a server issue of SC itself as like others also do.The limitations are really bad but SC is also a good one to watch vids as kao also said about the first load part of 5-6 MB = full speed etc.

Hhmmm,ok good idea to close the connection and re-connect it so I will try a little bit to add something like this.

PS: So I see the IDM tools is just a trail tool SameerRaj.Did you check whether its working with SC?The DownloadThemAll AddOn dosen't work on SC to DL faster and this addon says it has a accelerator but I see no better effect.

greetz

[LCF-AT]

Hi again,

so I checked this now what you said kao about closing connection after 5 MB and re-connect it.Do you know how to do this?Lets say I do "recv" the bytes and count this till 5 MB but if I now call closesocket API then I can't reconnect it again and I have to create a new socket which then starts again from the start.

Other question I have is how it should work if I wanna download a file not from the start.Lets say I did download a file what has a size of 30 MB and I did download 10 MB.So what is if I wanna download the rest of 20 MB?Some kind of resume function.Is there a way to set any kind of marker where to start the download etc?Any ideas?

greetz

[kao]

HTTP protocol has "Range" header. See my previous response and CURL commandline used. Another reference with examples: http://www.tutorialspoint.com/http/http_header_fields.htm (look for "Range")

 

[LCF-AT]

Hi again kao,

ah ok thanks,so I have not seen the Range paramter in your post.Ok I have test it a little and it does as you said. One thing about to to do it on that way to download with ranges.Lets say I do add a option in my little downloader tool to DL only 5 MB + reconnect to download next 5 MB etc till finished.Should I now only use & set static Range values like (Range: bytes=0-5000000) and as next (Range: bytes=5000000-10000000) etc (also if filesize has 7 MB for example) or should I count the bytes from recv API I got and stop the DL by me after 5 MB was reached and then using the added bytes for next range start?What is better etc?Do you have any clue how to handle this so easy as possible?

PS: The example video you found is gone now.I also tried to checkout a tool called JDownloader what also had a stop button + reconnect the download but there was no option to setup anything like this so that you have to stop / re-DL manually.Also after a while the downloader wasn't working anymore with that link but same link was still working to download or play directly in VLC player etc.Hmmm,strange tool.Anyway.

greetz

[Techlord]

Had a hard day today and was back just a while ago...

Nice to see that a lot of progress had been done on this thread

@kao : Nice work of digging into this issue deeper, that you have done yourself as well ! Great job !

As far as I was concerned, yesterday, I just wanted to check superficially and see definitively whether the restrictions were SERVER-SIDE or ISP-imposed. As I found out that they were server-side, I did not dig much deeper...

@LCF-AT : Actually, in MOST cases, the "trick" of stopping and re-starting the downloads, when the throttling is done server-side, does not consistently work for large files. Nearly all the major download managers (like IDM, DownThemAll, Jdownloader etc) have the PAUSE and CONTINUE options for downloads. Agreed, that even I use the PAUSE and then wait for a couple of seconds and then RESUME download, manually, when I urgently require to download files... But in many cases, if the ADMIN is good one , he'd configure the server such that the connect-pause-resume sequences work only a few times... Which is why, as you have discovered above with Jdownloader, the manual pause and resume did not work very well... Many a time, I whipped up a few quick scripts to get the job done, but very soon, the admin would wise up to it, and the "trick" would no longer work...

In any case, I would be very happy if we all could come up with a working script or solution that would consistently work with such servers that try to throttle our download speeds...

Greetz

 

[LCF-AT]

Hi again,

ah ok sounds like bad news for us to use a stop / resume method.Yes the JDownloader failed after a while with testing a DL on that manually way but the strange thing was that the DL was still working with other tools or also a direct access & play was also still possible.Seems that SC maybe checked for something to detect JD or whatever.So what is if I watch the video on the browser page and always seek forward or backward to the parts which are not loaded yet so this should be almost same access like to DL with Ranges xy.In that case the video should also just allow to just seek a few times only or?Hhmmm,so it seems that we can't really handle that issue as we thought.

So what is if we try to read first the entire filesize (100 MB for example) and do split them into 5 MB parts ranges (20 parts) and do download 5 parts at once and then next 5 parts etc and if we got all 20 parts on disc we put them all together and we got the entire file.I think other DL tools do working like this.If we got a problem during this DL then we use a proxy to DL go on etc.No idea whether the DL link we got depends on IP we use or not.

greetz

[LCF-AT]

Me again,

ok the partial download works but I got a problem with the file I got after so its broken.Seems that I do anything wrong or the request partial part starts on other position or something.

First I did request the file then check the header end where file bytes starts and this I write into file.Also I do count the bytes into QWord.Now after 5+ MB I change the current size I got in QW into dec byte string (Range: bytes=5123456-) and add this paramter into new request.Then I request the file again with new range and check again where the bytes starts at the end of Header response and write them into same file again.All in all its working so far but the file just runs fine for 30 seconds.Now I am not sure whether my another range requests are wrong or whether the response I got is wrong (maybe not exact the byte xy).Also it seems I have to set the last byte range too or it will download more and more also if the filesize is reached already.Hhmmmm.

Has maybe anyone a example how to handle this partial download method correctly?

greetz

[Techlord]

Hi @LCF-AT : Nice to know that you are progressing well. Can I give you a suggestion maybe, if you have not already done so ?

Have you tried reversing any Download Accelerator (DA) or looking at the Source Code of any Open Source download accelerators, to get a deeper understanding of how they 1) Break the file into multiple segments and then 2) download them in a RESUMABLE fashion (allowing PAUSE and RESUME) and then finally 3) Concatenating or JOINING all the segments together into a working SINGLE file ?

Nearly all the decent download accelerators can PAUSE a download session and continue it even after the system is re-started.

Maybe looking at all the APIs and the techniques that they use to achieve this would shed a lot of light  on our issue ?

Specifically, I think that we all can have a more detailed look into the mechanisms and techniques the DAs use to actually BREAK the file into segments, then download INDIVIDUAL segments in a way that there is NO OVERLAP of the file data between segments (which would later prove to be a problem when we join them again) and then JOIN the individually downloaded segments in such a way as to re-create the original file without overlaps of any data between segments.

For a start, since the DownThemAll! accelerator available as a Firefox addon is Open Source, and I believe that it would be an excellent idea to look into its source code ?

I will try to also have a look into reversing them sometime next week when I am free ... Right now, I am bogged down with work ...

Hope that @kao also can chip in when he's free ...

Greetz

P.S : @LCF-AT : No offence if you have already tried reversing and studying the Download Accelerators to look for the APIs and the specific techniques used ...

I had just merely given my "ideas" based on what I would do, when faced with a problem like this... Hope you did not take it in the wrong way ..

Quote

....No idea whether the DL link we got depends on IP we use or not.

Also, I have already confirmed in my earlier posts on this thread that the download link generated by the SC server is SPECIFIC for a particular IP and that it would throw a "Wrong IP" error if a download or PLAY is attempted from another different IP Address..

For example, in this link "http://stor23.streamcloud.eu:8080/6dv75xk3eooax3ptx3zynr7mvoeux2aa7pm7lrqzrham75df6emvm4uugm/video.mp4" the "6dv75xk3eooax3ptx3zynr7mvoeux2aa7pm7lrqzrham75df6emvm4uugm" is unique for a particular IP address ... Does not work for any other address...

Also, @LCF-AT : Now that the original file that we had been working on has been removed from the SC site, could you provide us with another link , so that we all can continue to work on it ?

Edited May 1, 2016 by Techlord

[LCF-AT]

Hi again,

no I didn't check any DA tools sources erc so far but today I still worked on that partial DL stuff and found the problem why it wasn't working for me yesterday (forgot to sub the header bytes) and fixed the problem and now its working. I updated my test download code and checked again to download normal and partial and below you can see the results in a picture I made...

....same file.Above you can see speed at end and DL time.Now I can save up over 5 minutes DL time for 71 MB.

Yes I thought already that the hash you get calculated by IP address (like other TV sites also do).Yes the testlink is gone.No idea why.So to find other test links of this site you can search for it (SC + xxx) or check any adult sites to find new links etc.

greetz

[Techlord]

Great to know @LCF-AT, that your issue is now resolved

By the way, I already searched (for a couple of mins only though) in Google using ".mp4 inurl:streamcloud" , among other search parameters, but probably it was a regional thing or something that I did not get any good hits...

But now that you were able to solve the matter yourself, it is no longer needed for us to look forthe links

Greetz

[LCF-AT]

Yes its nice to see that it works so far.I mean to save much time and getting full DL speed instead of this what we got before is a good step forward.Also nice to see that other DA tools like DTA & JD didn't do that or failed etc (advantage me). Anyway,I am just happy that something did work now also if I am no internet expert and just working with my own 2 cent brain but I also have the advantage to get many help & infos from people of this forum of course what brings the remaining 98 cent.

greetz