Chapter 3. Crawlers and Spiders

In this chapter, we will cover:

• Downloading a page for offline analysis with Wget
• Downloading a page for offline analysis with HTTrack
• Using ZAP's spider
• Using Burp Suite to crawl a website
• Repeating requests with Burp's repeater
• Using WebScarab
• Identifying relevant files and directories from crawling results