ArticleField.comInternet's Fastest Growing Free Content Article Directory

A problem faced by web data extractor software is similar to one faced by automatic phone dialers.

When automatic phone dialers first became available, their usefulness was limited due to the delay in receiving a dial tone when the phone line was initially engaged. By the time the line was ready, the dialer had already entered a few digits, causing it to consistently fail. A similar premature extraction can occur when scraping data from web sites.

To overcome this, a Wait Script is necessary when an AJAX Callback is unable to successfully retrieve data due to the webmaster’s use of a transitional word or phrase (typically a blinking “Loading…”) to tell the user the content is rendering. Visual Web Ripper confuses this for the actual data and extracts that unwanted text.

Remember good things come to those who wait. Follow these steps to have Visual Web Ripper extract the data you want.

Note: A working knowledge of OOP, especially in C# and VB.NET is EXTREMELY helpful.

To begin your Wait Script, navigate to your Options menu, click Action, click AJAX or javascript and click Script Wait Condition.

A window pops up, allowing you to manipulate premade code (either in C# or VB.NET). If the code is inaccessible, check Enable wait script.

Your code should look like this at the start:

1. public   static   bool  IsAjaxCallCompleted(WrWaitScriptArguments args)
2. {
3. try
4. {
5. //Place your condition code here.
6. //This example waits until the old content has been replaced with new content,
7. //but waits a maximum of 3 seconds.
9. if (args.SecondsWaited>3)
10. return   true ;
12. if (!args.OldContent.Equals(args.NewContent))
13. return   true ;
15. return   false ;
16. }
17. catch (Exception exp)
18. {
19. args.WriteDebug(exp.Message);
20. return   true ;
21. }
22. }

Be aware of some requirements with this solution:

• • All Wait Scripts must have one method (IsAjaxCallCompleted)
  • That method must have that exact name, so only change within the body.
  • The method must return true when the AJAX call has been completed and false otherwise.

The “loading” example would look like this (within the body):

1. try
2. {
3. if (args.SecondsWaited>3)
4. return   true ;
6. if (!args.OldContent.Equals(args.NewContent) && !args.NewContent.Equals( “Loading…” ))
7. return   true ;
9. return   false ;
10. }
11. catch (Exception exp)
12. {
13. args.WriteDebug(exp.Message);
14. return   true ;
15. }
16. }
17. }

Cascading Drop-Down Menus

A cascading drop-down menu is another hurdle that requires special handling when using Visual Web Ripper. Utilize this code to manipulate cascading drop-down menus:

1. try
2. {
3. if (args.SecondsWaited > 10)
4. return true;
6. if (!args.OldContent.Equals(args.NewContent) && !args.NewText.StartsWith(“[“))
7. return true;
9. return false;
10. }
11. catch (Exception exp)
12. {
13. args.WriteDebug(exp.Message);
14. return true;
15. }
16. }
17. }

Notes: A FormField element will be needed for every drop-down menu. VWR will wait for every AJAX element to change.

As web sites include an increased set of functionality to enhance user experience, this same functionality requires more powerful web data extraction software. Visual Web Ripper capably delivers in cases like this.

For more information about Best Web Scraping Software Please visit http://www.visualwebripper.com/

[!] Report This Article

Processing your request, Please wait....

Report as:	Broken Links Invalid / Wrong Links Spam Inappropriate Content Copyrighted Article Other
Your Name:
Your Email:
Message about this article: