Hello Everybody,

I'm trying to build one web crawler which will help user in finding and downloading specific contents from google. The main issue I'm facing i with compression of requests and responses. I have added gzip, dflat in request header so that I can compress the data but this thing is not working properly. I don't know why. In some of the system It's compressing the data but in some it will not.

I have tried first with PHP but its not working so I tried with asp.net but the results are same. Can anyone suggest how to overcome this issue?

The following is the webrequest call I have written in current asp.net page.

string Resp = "";
httpWebReq = (HttpWebRequest)WebRequest.Create("http://google.com");
httpWebReq.ServicePoint.UseNagleAlgorithm = false;
httpWebReq.ServicePoint.Expect100Continue = false;
httpWebReq.ServicePoint.ConnectionLimit = 6500000;
httpWebReq.ServicePoint.ConnectionLeaseTimeout = 50000;
httpWebReq.ServicePoint.MaxIdleTime = 500000;
httpWebReq.Connection = "keepalive";
httpWebReq.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip, deflate");
httpWebReq.UnsafeAuthenticatedConnectionSharing = true;
httpWebReq.AuthenticationLevel = System.Net.Security.AuthenticationLevel.None;
httpWebReq.ProtocolVersion = HttpVersion.Version11;
httpWebReq.Method = "GET";
rspn = (HttpWebResponse)httpWebReq.GetResponse();
response = rspn.GetResponseStream();
if (response.ContentEncoding.ToLower().Contains("gzip"))
{
   Resp = new StreamReader(new GZipStream(response,
   CompressionMode.Decompress), Encoding.UTF8).ReadToEnd();
}
else if (response.ContentEncoding.ToLower().Contains("deflate"))
{
    Resp = new StreamReader(new DeflateStream(response,
}
else
{
    Resp = new StreamReader(response, Encoding.UTF8).ReadToEnd();
}

Please suggest me something so that I can get the compressed data in all the system.

 

Thanks.

Hi,

not sure if this can be useful, in stackoverflow I found the same question, this is the reply 

http://stackoverflow.com/a/7775204/5822322

                      Dim responseStream As Stream = response.GetResponseStream()
                        If (response.ContentEncoding.ToLower().Contains("gzip")) Then
                            responseStream = New GZipStream(responseStream, CompressionMode.Decompress)
                        ElseIf (response.ContentEncoding.ToLower().Contains("deflate")) Then
                            responseStream = New DeflateStream(responseStream, CompressionMode.Decompress)
                        End If
                        Dim streamReader As System.IO.StreamReader = New System.IO.StreamReader(responseStream, Encoding.[Default])
                        Data = streamReader.ReadToEnd()
                      
                        responseStream.Close()
                        streamReader.Close()