visual studio .net general: WebClient and File Download -- visual studio .net general

RE: WebClient and File Download

doug
7/26/2007 6:38:03 AM

I'l try your code right now.

Here is the url i'm using: http://www.ffiec.gov/ratespread/YieldTable.csv

doug

[quoted text, click to view]

RE: WebClient and File Download

doug
7/26/2007 8:26:04 AM

I received:
The remote name could not be resolved: 'msdn2.microsoft.com'

We do have a proxy server with usererid crdentials.

doug

[quoted text, click to view]

RE: WebClient and File Download

doug
7/26/2007 4:50:00 PM

Still getting 404 errors when using synchronized calls. Async works but no
data, just file built size=0.

Here is my code:
Module Module1

Sub Main()

Try
Dim myWebClient As New WebClient()
Dim sBaseURL As String = "http://www.ffiec.gov"
Dim sURL As String =
"http://www.ffiec.gov/ratespread/YieldTable.csv"
'Dim sURL AS String = "http://msdn2.microsoft.com/default.aspx"
Dim sCredUid As String = "xxxxxxxx"
Dim sCredPwd As String = "yyyyyyyy"
Dim sCredDomain As String = "dddddddd"
Dim sProxyURL As String = "http://proxyconfig.sample.com"
Dim sProxyUID As String = "xxxxxxxx"
Dim sProxyPwd As String = "xxxxxxxx"
Dim sFile As String = "C:\Temp\1.txt"
Dim bUseAsync As Boolean = False ' toggle to test other methods

' Set Proxy Credentials
Dim myNetworkCredential As New NetworkCredential(sCredUid,
sCredPwd, sCredDomain)
myWebClient.Credentials = myNetworkCredential

' set Proxy
Dim myWebProxy As WebProxy
Dim myWebProxyCred As NetworkCredential
myWebProxyCred = New Net.NetworkCredential(sProxyUID, sProxyPwd)
myWebProxy = New System.Net.WebProxy(sProxyURL)
myWebProxy.Credentials = myWebProxyCred
myWebProxy.UseDefaultCredentials = True
myWebClient.Proxy = myWebProxy

' Set Uri
Dim myUri = New Uri(sURL)

' Get File
If My.Computer.FileSystem.FileExists(sFile) Then
My.Computer.FileSystem.DeleteFile(sFile)
End If

myWebClient.BaseAddress = sBaseURL
If bUseAsync Then
AddHandler myWebClient.DownloadFileCompleted, AddressOf
WebClient_DownloadFileCompleted
myWebClient.DownloadFileAsync(myUri, sFile)
Else
'myWebClient.DownloadFile(myUri, sFile)
Dim myByteArray As Byte() = myWebClient.DownloadData(sURL)
Dim myPageData As String =
System.Text.Encoding.ASCII.GetString(myByteArray)
End If

Console.Read()
myWebClient.Dispose()
Catch ex As Exception
Console.WriteLine(ex.Message)
Console.Read()
End Try
End Sub

Sub WebClient_DownloadFileCompleted(ByVal sender As Object, ByVal e As
AsyncCompletedEventArgs)
Console.WriteLine("Done")
End Sub

End Module

[quoted text, click to view]

RE: WebClient and File Download

doug
7/27/2007 1:54:01 AM

Walter,

I don't know anything about the proxy server except I had to request special
userid/password to use to get outside of it. Corporate security requirement.

The VB6/ITC code was snatched from MS site example years ago. Even you guys
used some ugly logic for proxy - complete with built in retries. This is the
part you want. Once this piece works, then the file get just sucks data into
it:
lOptionBufferLen = 0
sProgress = "HttpOpenRequest"
glHttpOpenRequestHndl = HttpOpenRequest(glInternetConnectHndl, "GET",
gsWebSite, "HTTP/1.0", vbNullString, 0, _
INTERNET_FLAG_RELOAD Or INTERNET_FLAG_KEEP_CONNECTION Or
SecFlag, 0)

If CBool(glHttpOpenRequestHndl) Then
sProgress = "HttpAddRequestHeaders1"
sHeader = "Accept-Language: en" & vbCrLf
iRetVal = HttpAddRequestHeaders(glHttpOpenRequestHndl, sHeader,
Len(sHeader), HTTP_ADDREQ_FLAG_REPLACE Or HTTP_ADDREQ_FLAG_ADD)

sProgress = "HttpAddRequestHeaders2"
sHeader = "Connection: Keep-Alive" & vbCrLf
iRetVal = HttpAddRequestHeaders(glHttpOpenRequestHndl, sHeader,
Len(sHeader), HTTP_ADDREQ_FLAG_REPLACE Or HTTP_ADDREQ_FLAG_ADD)

sProgress = "InternetSetOption1"
dwTimeOut = 420000 ' time out is set to 7 minutes
iRetVal = InternetSetOption(glHttpOpenRequestHndl,
INTERNET_OPTION_CONNECT_TIMEOUT, dwTimeOut, 4)
If iRetVal = 0 Then
Call C090_Logit(mcMODNAME, "M030_Prepare_For_Proxy_Connection",
sProgress & " iRetVal=" & iRetVal & " Err.LastDllError=" & Err.LastDllError &
" " & "INTERNET_OPTION_CONNECT_TIMEOUT", mCommon.gcAPPWARNING)
End If
sProgress = "InternetSetOption2"
iRetVal = InternetSetOption(glHttpOpenRequestHndl,
INTERNET_OPTION_RECEIVE_TIMEOUT, dwTimeOut, 4)
If iRetVal = 0 Then
Call C090_Logit(mcMODNAME, "M030_Prepare_For_Proxy_Connection",
sProgress & " iRetVal=" & iRetVal & " Err.LastDllError=" & Err.LastDllError &
" " & "INTERNET_OPTION_RECEIVE_TIMEOUT", mCommon.gcAPPWARNING)
End If
sProgress = "InternetSetOption3"
iRetVal = InternetSetOption(glHttpOpenRequestHndl,
INTERNET_OPTION_SEND_TIMEOUT, dwTimeOut, 4)
If iRetVal = 0 Then
Call C090_Logit(mcMODNAME, "M030_Prepare_For_Proxy_Connection",
sProgress & " iRetVal=" & iRetVal & " Err.LastDllError=" & Err.LastDllError &
" " & "INTERNET_OPTION_SEND_TIMEOUT", mCommon.gcAPPWARNING)
End If

Resend:
intCtr = intCtr + 1
sProgress = "HttpSendRequest"
If intCtr > 1 Then
Call C090_Logit(mcMODNAME, "M030_Prepare_For_Proxy_Connection",
"Invoking Resend for the " & intCtr & " time", mCommon.gcAPPINFO)
If intCtr >= giProxyRetries Then
' Abend
Call C090_Logit(mcMODNAME,
"M030_Prepare_For_Proxy_Connection", "ProxyRetries (" & giProxyRetries & ")
exceeded. Check for possible ProxyUID or ProxyPWD being wrong.",
mCommon.gcAPPERROR)
gbErrorEncountered = True
Exit Function
End If
End If
iRetVal = HttpSendRequest(glHttpOpenRequestHndl, vbNullString, 0,
sOptionBuffer, lOptionBufferLen)
If (iRetVal <> 1) And (Err.LastDllError = 12045) Then
'Certificate Authority is invalid.
Call C090_Logit(mcMODNAME, "M030_Prepare_For_Proxy_Connection",
"Invalid Cert Auth, resending", mCommon.gcAPPWARNING)
dwSecFlag = SECURITY_FLAG_IGNORE_UNKNOWN_CA
iRetVal = InternetSetOption(glHttpOpenRequestHndl,
INTERNET_OPTION_SECURITY_FLAGS, dwSecFlag, 4)
If CBool(iRetVal) Then
Call C090_Logit(mcMODNAME,
"M030_Prepare_For_Proxy_Connection", sProgress & " iRetVal=" & iRetVal & "
Err.LastDllError=" & Err.LastDllError & " " &
"INTERNET_OPTION_SECURITY_FLAGS", mCommon.gcAPPWARNING)
End If
GoTo Resend
ElseIf (iRetVal <> 1) And (Err.LastDllError = 12007) Then
'Could not resolve server name.
Call Sleep(60000)
lRC = DoEvents()
Call C090_Logit(mcMODNAME, "M030_Prepare_For_Proxy_Connection",
"Name could not be resolved, resending", mCommon.gcAPPWARNING)
GoTo Resend
End If

If CBool(iRetVal) Then
Dim dwStatus As Long, dwStatusSize As Long
dwStatusSize = Len(dwStatus)
sProgress = "HttpQueryInfo"
HttpQueryInfo glHttpOpenRequestHndl, HTTP_QUERY_FLAG_NUMBER Or
HTTP_QUERY_STATUS_CODE, dwStatus, dwStatusSize, 0
If bSkipServiceId Then
Call C090_Logit(mcMODNAME,
"M030_Prepare_For_Proxy_Connection", "ServiceId is set to NONE, bypass
setting userid/password", mCommon.gcAPPWARNING)
Else
Select Case dwStatus
Case HTTP_STATUS_PROXY_AUTH_REQ
iRetVal = InternetSetOptionStr(glHttpOpenRequestHndl,
INTERNET_OPTION_PROXY_USERNAME, _
strProxyUID, Len(strProxyUID) + 1)
iRetVal = InternetSetOptionStr(glHttpOpenRequestHndl,
INTERNET_OPTION_PROXY_PASSWORD, _
strProxyPWD, Len(strProxyPWD) + 1)
GoTo Resend
Case HTTP_STATUS_DENIED
iRetVal = InternetSetOptionStr(glHttpOpenRequestHndl,
INTERNET_OPTION_USERNAME, _
strProxyUID, Len(strProxyUID) + 1)
iRetVal = InternetSetOptionStr(glHttpOpenRequestHndl,
INTERNET_OPTION_PASSWORD, _
strProxyPWD, Len(strProxyPWD) + 1)
GoTo Resend
End Select
End If

' Good - Past Proxy
blnRC = True

Else
' HttpSendRequest failed
Call C090_Logit(mcMODNAME, "M030_Prepare_For_Proxy_Connection",
"HttpSendRequest call failed; " & sProgress & " - iRetVal=" & iRetVal & "
Err.LastDllError=" & Err.LastDllError & ".", mCommon.gcAPPERROR)
End If
Else
' HttpOpenRequest failed
Call C090_Logit(mcMODNAME, "M030_Prepare_For_Proxy_Connection",
"HttpOpenRequest call failed; " & sProgress & " - Error code: " &
Err.LastDllError & ".", mCommon.gcAPPERROR)
End If

[quoted text, click to view]

RE: WebClient and File Download

wawang NO[at]SPAM online.microsoft.com
7/27/2007 2:42:38 AM

Hi,

Thanks for the code.

You once mentioned that accessing the root url succeeds, only the .csv file
failed to download, is that still the case?

It now seems related to the proxy server setting. The web site
"http://www.ffiec.gov" can be accessed from my side, therefore I don't
think you will need specify myWebClient.Credentials; instead, I think all
you needed is to setup the Proxy correctly.

Could you please tell me more about the proxy you're using? For example,
are you using Microsoft ISA Firewall client? How's your IE's proxy setting?
If you're able to access the .csv file in IE but returns 404 error in code,
I'm suspecting it's because the proxy server is not correctly configured.
However, I cannot see any obvious issues from your code.

You also mentioned that your VB6 code using ITC works correctly, could you
please tell me how's the VB6 implementation of downloading the same file?
Thanks.

Regards,
Walter Wang (wawang@online.microsoft.com, remove 'online.')
Microsoft Online Community Support

==================================================
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
==================================================

This posting is provided "AS IS" with no warranties, and confers no rights.

RE: WebClient and File Download

doug
7/27/2007 10:34:02 AM

Additionally, I get the 404 error bringing up pages I built on our intranet
inside the firewall.

[quoted text, click to view]

RE: WebClient and File Download

doug
7/27/2007 3:44:00 PM

I made a few tweaks to remove proxy stuff and I'm able to retrieve files on
our intranet - just issue getting outside firewall now giving 404 errors.

doug

[quoted text, click to view]

RE: WebClient and File Download

wawang NO[at]SPAM online.microsoft.com
7/30/2007 12:00:00 AM

Hi,

If you use IE to access web sites outside the firewall, what's the proxy
setting in IE?

I noticed you're using code:

myWebProxy = New System.Net.WebProxy(sProxyURL)

This means you're using the URL for automatic proxy configuration, please
check if this is the correct proxy setting and if the URL exists or not: if
this URL doesn't exist, I suspect this 404 error is from this (though I
haven't verified it).

Normally we use a host and a port to construct the proxy:

myWebProxy = new System.Net.WebProxy("192.168.1.1", 8080)

The correct proxy setting is the key here to fully troubleshoot the issue.

Another thing to try is to debug the VB6 code and see what's the required
information that it's needed to access the proxy.

Regards,
Walter Wang (wawang@online.microsoft.com, remove 'online.')
Microsoft Online Community Support

==================================================
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
==================================================

This posting is provided "AS IS" with no warranties, and confers no rights.

RE: WebClient and File Download

doug
7/30/2007 6:02:01 AM

IE Proxy vsettings are as you indicated, aitomatic configuration, just the
URL, no port. There used to be a port many many yearts ago with a different
URL, but this is what has been working for at least 2 years.

I know the MS code I downloaded expected issues and built in retries. It
might take a few days, but I'll debug it.

[quoted text, click to view]

RE: WebClient and File Download

wawang NO[at]SPAM online.microsoft.com
7/31/2007 12:00:00 AM

Hi,

Please try WebProxy.GetDefaultProxy:

#WebProxy Class (System.Net)
http://msdn2.microsoft.com/en-us/library/system.net.webproxy(vs.80).aspx
<quote>
The GetDefaultProxy method returns an instance of the WebProxy class with
the Address, BypassProxyOnLocal, and BypassList properties set to the
values used by Internet Explorer 5.5 and later.
</quote>

If your IE is working correctly, this should be able to return the correct
proxy.

Regards,
Walter Wang (wawang@online.microsoft.com, remove 'online.')
Microsoft Online Community Support

==================================================
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
==================================================

This posting is provided "AS IS" with no warranties, and confers no rights.

RE: WebClient and File Download

doug
7/31/2007 1:42:01 PM

I'm usingh Framework v2. This gives me the "depreciated msg".

doug

[quoted text, click to view]

RE: WebClient and File Download

wawang NO[at]SPAM online.microsoft.com
8/1/2007 2:09:30 AM

Since I don't have the exact proxy server as yours, it's difficult for me
to test the code first on my side before giving suggestions. Sorry about
that.

After inspecting the code of WebProxy using Reflector
(http://www.aisto.com/roeder/dotnet/) , I can see it's equivalent when
you're using the WebProxy(Uri) constructor, which uses Web Proxy Auto
Discovery protocol (WPAD).

Could you please send me your VB6 code? If we really cannot use the
WebClient here, we might be able to use the same WinInet API in VB.NET via
P/Invoke.

Regards,
Walter Wang (wawang@online.microsoft.com, remove 'online.')
Microsoft Online Community Support

==================================================
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
==================================================

This posting is provided "AS IS" with no warranties, and confers no rights.

RE: WebClient and File Download

doug
8/31/2007 9:32:02 AM

Walter,

If you recall I was able to use WebClient to access files inside firewall
easily and cleanly using .Net Framework classes. I wasn't able to use these
same classes to get outside the firewall using a "service id" credentials
that were different from my own.

I returned to my version of code using WinNet functions and used the
InetsCtlsObjects and reworked the declares and invokations to use Integers
and IntPtr as needed and slowly got each piece working. Still getting
warnings on HttpAddRequestHeaders but its working. File arrives just fine on
my machine in debug mode.

Thank you verry much for your time and advice.

doug

[quoted text, click to view]

RE: WebClient and File Download

doug
8/31/2007 10:04:01 AM

Not sure why I thouight I would dodge this error: 80040112 on trying to
invoke InetCtrlsObjects. I underrstand this is a licensing issue? WebClient
isn't working. I cannot continue to use VB6 IDE to support this code. I
need to use VB.Net and this VB6 supplied object.

I currently have MSVSv6 and MSVS .Net 2005 installed on my PC. Runs okay.

Program does NOT run okay ("80040112") when copied to server (and
Interop.InetCtlsObjects.dll copied to same execution folder as EXE).

[quoted text, click to view]

RE: WebClient and File Download

wawang NO[at]SPAM online.microsoft.com (
9/3/2007 2:59:35 AM

Hi Doug,

You may want to try to merge the registry key in your development machine's
%windir%\system32\inetctls.srg which has following content:

[HKEY_CLASSES_ROOT\Licenses\78E1BDD1-9941-11cf-9756-00AA00C00908]
@ = "yjrjvqkjlqqjnqkjvprqsjnjvkuknjpjtoun"

However, the ITC control in VB6 is a wrap on WinInet APIs, I can see
actually you've already using some WinInet APIs directly to setup the
authentication/proxy, you're simply using ITC control's download function,
which actually can be also replaced with some WinInet API.

Please refer to following KB for more info and post here when you have
questions to migrate the ITC control calls into WinInet apis calls in
VB.NET:

#INFO: List of WinInet API FTP Samples
http://support.microsoft.com/kb/289253

Regards,
Walter Wang (wawang@online.microsoft.com, remove 'online.')
Microsoft Online Community Support

==================================================
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
==================================================

This posting is provided "AS IS" with no warranties, and confers no rights.

RE: WebClient and File Download

doug
9/6/2007 6:28:01 AM

The merging registry entry idea to resolve the ITC license issue on .Net is
somewhat appealing, but not something our production environment is going to
embrace.

I did spend dome time replacing the ITC class with InternetReadFile. As you
stated I had already put most of the front end WinInet calls in place to get
past the proxy settings. What I see in my testing is all the calls appear to
be working. The added InternetReadFile is returning 0 bytes on read.

[quoted text, click to view]

RE: WebClient and File Download

wawang NO[at]SPAM online.microsoft.com (
9/7/2007 2:18:55 AM

Hi Doug,

I haven¡¯t gone through all the code, however, per the question about
InternetReadFile, I do see some issues:

Public Declare Function InternetReadFile Lib "wininet.dll" _
(ByVal hFile As Integer, _
ByVal sBuffer As String, _
ByVal lNumBytesToRead As Integer, _
ByVal lNumberOfBytesRead As Integer) As Integer

The MSDN library show its signature as:

BOOL InternetReadFile(
HINTERNET hFile,
LPVOID lpBuffer,
DWORD dwNumberOfBytesToRead,
LPDWORD lpdwNumberOfBytesRead
);

We have some correction to make to the P/Invoke declaration:

* LPVOID lpBuffer: means a void memory pointer, we will need to use a byte
array (with size of lNumBytesToRead)
* LPDWORD lpdwNumberOfBytesRead: this means a pointer to a DWORD, we can
use ¡°byref lNumberOfBytesRead as integer¡±.
* We need to turn on SetLastError switch for the declaration so that we
can trap the error code if it fails (again, the MSDN library should tell
you that it supports GetLastError)

The correct declaration of this API should be:

'InternetReadFile
<DllImport("WinInet.dll", EntryPoint:="InternetReadFile",
CharSet:=CharSet.Auto, SetLastError:=True)> _
Public Shared Function InternetReadFile(
ByVal handle As Int32, _
<MarshalAs(UnmanagedType.LPArray)> _
ByVal newBuffer() As Byte, _
ByVal bufferLength As Int32, _
ByRef bytesRead As Int32) As Int32
End Function

For a more complete working example, please see following resource:

#WinInet vs. Managed Options: ASP Alliance
http://aspalliance.com/543_WinInet_vs_Managed_Options

Hope this helps.

Regards,
Walter Wang (wawang@online.microsoft.com, remove 'online.')
Microsoft Online Community Support

==================================================
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
==================================================

This posting is provided "AS IS" with no warranties, and confers no rights.

RE: WebClient and File Download

doug
9/14/2007 10:02:02 AM

Walter,

I tweaked a few more of the APIs, added call to InternetGetLastResponseInfo,
and did more testing until I got my pgm to work locally on my workstation
using WinInet.DLL calls exclusively.

When I moved program to server it failed to find InternetGetLastResponseInfo
entrypoint in WinInet.dll. My wininet.dll on workstation is dated 4/18/2007.
Server version is 4/20/2007. All other WinInet.dll files are in package
directories on server in source code libraries. But I'm guessing server
version is finding a WinInet.dll old enough to not have the new method
declaration.

I changed program to issue GetLastError() and received a 12002 (Timeout).
Found KB article q176420 that says InternetSetOptions will fail on setting
selected timeout values on pre-IE6 version (my words). So again, I'm
thinking my program when running on server is finding an earlier version of
WinInet.dll.

doug

[quoted text, click to view]

Got something to say? Click here to post a reply to this thread.

Don't see what you're looking for? Try a search.

View other messages in the visual studio .net general group.