Every now and then you may end up in a situation where SharePoint crawler returns just one top level error or multiple errors. Probably most of us would first enable verbose logging for SharePoint search, run crawl and monitor ULS logs with ULS Viewer. This is a good beginning, but it often offers very little information and very limited insight to crawler’s behavior.

By using Fiddler as a proxy server for SharePoint crawler, we can actually track what the crawler is actually crawling. This is very handy when debugging problems arising e.g. from multiple redirects, proxy- or authentication issues. You can enable fiddler proxy by following the following steps:

1. 1. Download and install Fiddler2
2. 2. Start Fiddler2 to enable it to act as proxy
3. 3. Create file called mssdmn.exe.config to folder %CommonProgramFiles%\Microsoft Office Servers\14.0\Bin (mssdmn.exe file should be also found here)
4. 4. Add following lines to the file
   <configuration>
   <system.net>
   <defaultProxy>
   <proxy
   usesystemdefault="true"
   proxyaddress="http://127.0.0.1:8888"
   bypassonlocal="true"
   />
   </defaultProxy>
   </system.net>
   </configuration>
5. 5. Restart SharePoint Server Search 14 service on server to apply proxy settings.
6. 6. Start SharePoint crawl and monitor Fiddler2. You should see something similar to picture below:
   

So, now you have a little bit better insight to SharePoint’s crawler. Hope this helps you to solve crawler related problems.

Update: You need to run Fiddler has the same account the Search Service is running. Otherwise Fiddler doesn’t log anything as it will run in a different user context.

Popularity: 3% [?]