advanced log file analysis for crawl errors detection