Jump to content
Tuts 4 You

Why GZip Dflat compression not working in all system


kate

Recommended Posts

Hello Everybody,

I'm trying to build one web crawler which will help user in finding and downloading specific contents from google. The main issue I'm facing i with compression of requests and responses. I have added gzip, dflat in request header so that I can compress the data but this thing is not working properly. I don't know why. In some of the system It's compressing the data but in some it will not.

I have tried first with PHP but its not working so I tried with asp.net but the results are same. Can anyone suggest how to overcome this issue?

The following is the webrequest call I have written in current asp.net page.

string Resp = "";
httpWebReq = (HttpWebRequest)WebRequest.Create("http://google.com");
httpWebReq.ServicePoint.UseNagleAlgorithm = false;
httpWebReq.ServicePoint.Expect100Continue = false;
httpWebReq.ServicePoint.ConnectionLimit = 6500000;
httpWebReq.ServicePoint.ConnectionLeaseTimeout = 50000;
httpWebReq.ServicePoint.MaxIdleTime = 500000;
httpWebReq.Connection = "keepalive";
httpWebReq.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip, deflate");
httpWebReq.UnsafeAuthenticatedConnectionSharing = true;
httpWebReq.AuthenticationLevel = System.Net.Security.AuthenticationLevel.None;
httpWebReq.ProtocolVersion = HttpVersion.Version11;
httpWebReq.Method = "GET";
rspn = (HttpWebResponse)httpWebReq.GetResponse();
response = rspn.GetResponseStream();
if (response.ContentEncoding.ToLower().Contains("gzip"))
{
   Resp = new StreamReader(new GZipStream(response,
   CompressionMode.Decompress), Encoding.UTF8).ReadToEnd();
}
else if (response.ContentEncoding.ToLower().Contains("deflate"))
{
    Resp = new StreamReader(new DeflateStream(response,
}
else
{
    Resp = new StreamReader(response, Encoding.UTF8).ReadToEnd();
}

Please suggest me something so that I can get the compressed data in all the system.

 

Thanks.

  • Like 1
Link to post

                      Dim responseStream As Stream = response.GetResponseStream()
                        If (response.ContentEncoding.ToLower().Contains("gzip")) Then
                            responseStream = New GZipStream(responseStream, CompressionMode.Decompress)
                        ElseIf (response.ContentEncoding.ToLower().Contains("deflate")) Then
                            responseStream = New DeflateStream(responseStream, CompressionMode.Decompress)
                        End If
                        Dim streamReader As System.IO.StreamReader = New System.IO.StreamReader(responseStream, Encoding.[Default])
                        Data = streamReader.ReadToEnd()
                      
                        responseStream.Close()
                        streamReader.Close()

Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...