Must have More Modules for Eugene Kalinin

Can be used to pull out data from the robots.txt file and feed it to other applications or statistics tools.Will extract target URL, reason for access and status data.What is new in this release:Handles http(s) request errors. What is new in version...