How do I troubleshoot SharePoint? So many logs!

 It's true there are likely more logs in SharePoint than any other Microsoft application (if you don't count the apps built on top of it) :)

On the Front End

  • Troubleshooting Web Parts - These errors often don't get logged or even seen on a slow loading page, but you can see what's going on in the web part maintance page and delete the offending parts.  There's a Ted Pattison Feature called SharePoint Debugger for this (turn it on right from site actions).
  • Verbose page errors - turn on debug mode in the web.config.  There is actually a trace log provider with MSDN samples so you can write your logs to this.
  • IIS Logs - everything from server error numbers, client error numbers, usernames, session info, bytes in, bytes sent, a bit archaic if you don't know what you're looking.  Best free tool - Microsoft IIS Log parser: runs like a SQL query tool against the logs
  • SharePoint ULS (United Logging Service) logs - These can be ugly, but they often are the only place that contains some of these errors.  If you're having problems with features, activation, web parts, any timer jobs like profile import user info sync, and really that SharePoint goo, then this is definitely a place where you just have to be familiar with.  Most of the work is searching for "error," "fail," "failed."  You can crank up the logging through the SharePoint Central Admin logging Central Admin: Central Administration > Operations > Diagnostics Logging.  Find these logs at C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\LOGS.  Scott Hilier put up a codeplex feature called SharePoint "log viewer" that allows you to view these logs in Central admin.  Looks like he also has some stuff on diagnostics up there.  Also the SharePoint Ninja (new to me) has written a SharePoint ULS log parser.
  • IIS Worker Process Logs - I don't know many people who spend a lot of time here, but I've found most of my memory issues are manifested here.  You can see a lot of what's loading up and what happens before the worker process cycles.  They aren't that hard to read, but there is a lot of noise.  All of these logs have lots of noise.  Use the Trace stuff in IIS 7... it's awesome for getting to the bottom of issues that manifest themselves in the worker processes.
  • Application (Event) Log - My favorite place to spend time when funny things are happening on a server.  Most of the work is filtering to find the time when the failure occured, walking backward to see what led up to the failure.  There are some huge docs on Verbose WSS errors  and MOSS events.
  • System (Event) Log - Memory and system events mostly around hardware failures, OS failures, and on and on.  Don't forget this one.  IIS often has errors here.  Event viewer is the most common place this is viewed.
  • Security (Event) Log - You'll find a lot of the authentication stuff seems to end up here.   If you're doing Kerberos, you can't live without this log. 
  • All those user specific logs in %temp% logs- SharePoint Installation and Upgrade logs (version to version and build to build and hotfix, config wizard) Logs (also anything STSADM spits out in a log like an stsadm solution deployment) - If you're having problems with upgrade and you ignore these logs... troubleshooting will take 10X as long to troubleshoot maybe longer.  These logs have tons and tons of noise.  You basically have to find where the failure occurred and again walk backward.  Searching for "fail" or "error" in a text editor is where a lot of us have lived.  The SPS 2003 to MOSS logs are at C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\Logs\Upgrade.log
  • Performance Logs (generated by perfmon) - performance logs aren't logged unless you turn them on, but man I bet I use them 100% of the time if I'm troubleshooting Memory, CPU, Disk and Network issues
  • Netmon Logs (generated by NetMon) - again not turned on by default, but if you're trying to trace between the client and the server a good fiddler2 session or visual studio test team or classic network monitor from the SMS bits is going to help you understand what's coming in from your DCs, other member servers, SMTP, and on and on.
  • Gatherer/Crawl Logs - If you're troubleshooting Indexing you'll be going a few places, you'll see gatherer logs are in the database but there's an interface in the SSP admin UI.  There's also the Change Log, but it isn't something you'll use much.

Backend Logs

  • SQL Logs - these are truly underrated.  There is so much *good* information in the SQL Logs.  I've found problems with memory, settings and configuration, things that expose needed hotfixes.  The whole -g startup switch on 32 bit systems for extended memory addressing was only exposed by drilling into the logs and seeing the "out of memory" errors.  The SQL team does a good job of helping you track error numbers to problems to then solutions.
  • SQL Profiler Logs - You have to know SQL profiler to truly troubleshoot backend issues.
  • System, App, Security (watch for kerb issues which require SPN), Netmon, Perfmon - see above

 

SharePoint Administration Suite

With this quick overview, you can see there's just so much here.  Have you tried out the Bamboo Solutions SharePoint System Log Manager?  It was designed to be a central interface for troubleshooting logs.  I haven't either, but I plan to give you a review, but wanted to first tell you what's out there so you can see I'm not biased.

"System Log Manager gives your SharePoint administrators easy, customizable access to the vast library of system logs that are scattered throughout your system. Empower administrators to sort through Windows, SQL, IIS, and SharePoint logs from one central interface, creating customized reports and filtering out irrelevant data without making complex system calls or relying on a command line."

I'll follow up with a review...  Let me know what your experience has been with it.

Joel Oleson - SharePoint Joel

Other Good Resources:

To turn on netlogon.logging see: Enabling debug logging for the Net Logon service http://support.microsoft.com/default.aspx?scid=kb;EN-US;109626 .

264921  How IIS authenticates browser clients

http://support.microsoft.com/default.aspx?scid=kb;EN-US;264921

907273  Troubleshooting HTTP 401 errors in IIS

http://support.microsoft.com/default.aspx?scid=kb;EN-US;907273

Remote Debug Viewing
http://msdn.microsoft.com/fr-fr/library/h0hfz6fc(VS.85).aspx

Troubleshooting Common SharePoint Errors

http://support.microsoft.com/kb/944267

Update: Also dont forget SPTraceView by Hristo Pavlov (Thanks Anders for the comment)

"Because SPTraceView processes the tracing in real time you can identify errors and events as they happen. That is as soon as you interact with the SharePoint GUI when testing/debugging your custom SharePoint solutions including web parts, event receivers, workflows and all other SharePoint technology components.

SPTraceView is a very light weight software. It is a single 88kb executable file that doesn’t need to be installed and will work simply after you copy it to your machine. It can run in a farm and can provide the trace events from individual servers in the farm to a central location where you can monitor them in real time. It can also save the events of your particular interest in it’s own XML log files which you can review later. It can be also very useful to administrators for determining how healthy their SharePoint farm is."

(Another slick tool.  I can totally see using this in code reviews and test cycles.  Thanks...)


Posted Sep 29 2008, 06:06 AM by Joel Oleson

Comments

Anders Rask wrote re: How do I troubleshoot SharePoint? So many logs!
on Tue, Sep 30 2008 9:28 AM

Excellent overview!

Also dont forget SPTraceView by Hristo Pavlov!

It pipes the ULS trace messages (all of them! not just the ones that end up in the log file!) "on the fly" either to a log file, or (my favourite) to DebugView. You can filter on process names (eg. stsadm.exe), event levels, categories etc etc.

A must have tool for debugging especially assemblies you dont "own", or for debugging solution deployment.

hristopavlov.wordpress.com/.../sptraceview-lightweight-tool-for-monitoring-the-sharepoint-diagnostic-logging-in-real-time

Oksana P wrote re: How do I troubleshoot SharePoint? So many logs!
on Fri, Oct 10 2008 11:43 AM

Any recommendations on what to do with the logs in "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\Logs"?  After a while, that directory takes up a lot of room.  Are there any best practices on whether this information should be dumped or saved?

Add a Comment

Please sign into Bamboo Nation to leave a comment.

About Joel Oleson

As Sr. Technical Product Manager for SharePoint Products and Technologies at Microsoft, I owned the technical messaging and product positioning for the IT Professional audience. With more than 12 years of experience in the Internet Industry, I have a vast background in Web technologies. First, starting out with Internet and intranet application web design including enterprise backend data storage. I later moved into design and infrastructure architecture, engineering, and operations. I'm now a community evangelist for SharePoint and working to help companies solve their IT issues related to deployment and governance.

Blogs

    The Bamboo Team Blog
  • Home

Proud Media Sponsor of:

Subscribe by Email

Syndication

Bamboo Nation Almost Everywhere

Bamboo Solutions on Facebook

Bamboo Solutions on Google+

Bamboo Solutions on LinkedIn

Bamboo Solutions on YouTube

Bamboo Now in Alltop!

Featured in Alltop

Bamboo Solutions Corporation, 2002-2014