I have created a program that screen scrapes about 50 sites. Currently, when the user comes to the site at 3am and 6am or at 3pm and 6pm the program scrapes the 50 sites and updates the database with the newly scraped information. Once all the information is updated, it is displayed on the default.aspx page. As you might expect, it takes a long time for the page to load during this process. I am looking for a way to reduce the time. What is the best way to do this? Is it possible to program a software to execute itself at a certain time without the initial request from the browser? I think this is not possible due to the rules of the client server relationship. Any insight would be appreciated. Thanks.
...
Go to the complete details ...