SharePoint: How to prevent your content from being crawled

A client of mine did not want parts of their Intranet based on MOSS 2007 to be searched; now that’s quite easy to do with Crawl rules. As they wanted strict control over the content that can be crawled I trained them on how to control ‘the crawl’ behaviour for content within SharePoint.

There are three places within SharePoint where end users can change search visibility setting or exclude content for search.

1. Change Visibility for a Web site: In the Site Settings page, site owners can click the Search Visibility link to go to the Search Visibility page SharePoint. In the Indexing Site Content group, selecting the No option to will exclude all content within the site. The crawler will in turn simply skip the site and not include any of its content in the index.
2. Exclude Site Columns: In the Site Settings page, site owners can click the Searchable Columns link to go to the Search Settings for Fields page in SharePoint. This page will enumerate all the site columns defined at the current site level. Selecting the NoCrawl check box for the site columns will exclude them from future crawls.
3. Exclude Lists: The last option for excluding content exists at the level of the SharePoint list settings. It enables list owners to exclude the list and all the content within it from crawls. In the settings page of the list, click the Advanced Settings link. Selecting the No option under Search Available for crawl tells the crawler to exclude the specific list from a crawl.

Also if there are certain pages or entire site content that site designers don’t want to be crawled, simply putting the no index directive in the head section of the master page or layout page’s mark-up will stop the SharePoint crawler from crawling the pages:

<META NAME=” ROBOTS” CONTENT=” NOHTMLINDEX”/>

Why do my changes not show in other sites?

A detailed look at how the Sync works

I was asked recently to advise on how to configure a MOSS 2007 installation to allow end users to update their display name as they had previously been able to do in 2003. My initial thoughts on this were that its a simple update of the profile properties to allow them to edit the Name field, however the results achieved in testing appeared inconsistent.

In this article I will show

• How to setup the profile properties
• What the user sees and can edit
• What services/timer jobs are involved in the Sync and how to speed this up for testing
• The conditions that need to be met to trigger the profile information to be replicated.

Setting Up Profile Properties

The first step is to setup the profile properties. In this example I will be looking at enabling users to edit the display name (”PreferredName”) field. This example assumes that you have MOSS 2007 and a Shared Service Provider has been setup and configured.

From Central Admin choose the Shared Service Provider from the left hand menu. This should display a screen similar to the one below.

Select the User profiles and properties option as highlighted. This will allow you to add new properties or as in this example edit an existing property.

Scroll to the bottom of the page and click View profile properties. On the list of properties select the Name field and choose Edit.

The profile properties page allows you to configure the profile property. Ensure that the field selected is the PreferredName field, this has default Display Name of Name.

Check that the field is checked for replication. You will note that the text suggests that properties that can be replicated cannot be edited by the user, however this does not appear to be the case.

Choose Allow users to edit values for this property, this will allow the user to change the value via MySite.

Change the mapping to ‘not mapped’. If you do not change this any modification the user makes will be overwritten at the next profile import.

Note: When a user firsts accesses a site the User Info is extracted directly from AD (or your provider, not tested) and it still pulls the AD based display name into this field as it is not related to the profile mapping.

We will look at the different states for the User Info and profile imports below.

The properties page should show the PreferredName field is no longer mapped.

That is all of the changes required to make the display name editable by the user.

What The User Sees and Can Edit

When the user navigates to a SharePoint site the initial view will show the information contained in AD (I assume this is the same for other providers but you will need to test) including the display name.

AD user with defined Display Name, this will be used initially when the user accesses a SharePoint site as shown below.

To edit the display name navigate to your My Site, this will initially look like the image below, notice that the Welcome message is not based on the display name from AD but the users account name. In this example I have blurred the domain name.

Click on the Details option under the My Profile Quick Launch and edit the Name field.

In order to test this I have used a time stamp so I could repeat the tests and identify what had changed.

Clicking on Save will take you back to the My Site home page and will show some of the fields have updated but others have not. The fields that have not updated are based on the User Info table and is updated by the Quick Profile Synchronization timer job (see below).

The Quick Profile Synchronization timer job is scheduled too run regularly (minutes) but I have found in testing, and this will increase as your number of users and sites increase, it can take a while for the changes to take place. You can force this timer job by running the STSADM operation SYNC.

stsadm -o sync

Note: this starts the job, the changes will not appear instantly.

After the sync has been performed the User Info table will be updated with the values from the My Profile details.

Why do some Team Sites not update with my details

The main reason for confusion around the updating (sync’ing) of My Profile details into team sites is that it is now based on if the user is Active. As part of the upgrade to V3 Microsoft introduced the idea of users being active in a site rather than just having access.

When you first visit a site a record is recorded in the UserInfo table of the sites Content Database. The field tp_IsActive is defaulted to false, which means you have visited but have not interacted with the site.

The Sync process uses this value to determine if the user info should be replicated from the My Profile details as edited above.

Following on from the example above you should be able to refresh your team site that you had only visited and see the user name is still the same, it was not updated when the My Site was updated.

If you now interact with the site, either add a document or edit a link, in effect become an active user of the site, and then run the stsadm -o sync command as above the My Profile changes to the user name will be replicated to the site.

Sync Timer Jobs and STSADM

The synchronisation of the profile properties and information are based on two timer jobs, and one stsadm command (as used above). These jobs are per web application so you may see multiple of these configured in your environment. Access to these is via Centra Admin -> Operations.

These timer jobs are part of the Microsoft.Office.Server.UserProfiles namespace and specifically

WSSProfileSynch

Handles the synchronization of user profile data in the Windows SharePoint Services user information list on each site, and the synchronization of Windows SharePoint Services members group membership in the user profile memberships. This class is not intended to be used directly from your code; use stsadm.exe instead.

WSSSweepSynch

Handles the incremental synchronization of user profile data in the Windows SharePoint Services user information list on each site. This class is not intended to be used directly from your code; use stsadm.exe instead.

I have yet to determine exactly which of these processes are triggered using the stsadm -o sync command.

Additional Information

Profile Synchronisation on MSDN

Sahil Malik has a good post on the high level information flow within MOSS here.

I have also included below some sample code that I used to try and speed up the timer jobs. You may want to use these to help understand what each timer job does. Note, changing the Profile Synchronization schedule will not remain, another time job appears to reset this to the installed defaults.

static void Main(string[] args){ List<System.Uri> urls = new List<System.Uri>(); urls.Add(new System.Uri(“http://mysite_webapplication”)); urls.Add(new System.Uri(“http://portal_webapplication”)); for (int i = 0; i < urls.Count; i++) { SPWebApplication webApp = SPWebApplication.Lookup(urls[i]); foreach (SPJobDefinition job in webApp.JobDefinitions) { // “profsynch” = Profile Syncronisation // “sweepsync” = Quick Profile Syncronisation if (job.Name == “profsynch”) { SPMinuteSchedule schedule = new SPMinuteSchedule(); schedule.BeginSecond = 0; schedule.EndSecond = 59; schedule.Interval = 1; job.Schedule = schedule; job.Update(); Console.WriteLine(“Temporarily Updated Profile Sync for Web Application {0}”, webApp.Name); } } } Console.ReadLine();}

Error and Omissions

Warning: Access to the database directly is not supported, I performed this in my test environment to understand why things worked the way they did.

Using a famous Todd Bleeker line from TechEd Orlando – I reserve the right to be wrong! This is what I have found so far, but my knowledge and understanding of this may change over time.

Everything about SharePoint UserProfiles

There are two kinds of User Profile information stored within SharePoint. One is WSS Profile which gets stored in UserInfo table in the content database for each site collection.The other is MOSS Profile which gets stored in SSP database and that gets sync with AD at regular intervals by Profile Synchronisation job. There are two profile sync jobs

1. The profsynch (Profile Synchronization) timer job which runs every hour
2. The sweepsync (Quick Profile Synchronization) timer job which runs each minute

To run the Profile sync jobs yourself you can execute below command.
stsadm -o sync

-excludewebapps <Web applications>

-synctiming <schedule>

-sweeptiming <schedule>

-listolddatabases <days>

-deleteolddatabases <days>

-IgnoreIsActive <0/1>

Quick Launch Bar Navigation Missing From Web Part Pages in SharePoint

It's rather disturbing that the Left Side Navigation (Quick Launch) by default, does not display on new web part pages created in SharePoint. There are a number of workarounds on the web which include hacking each newly created page, or hacking an existing document template.

I like to use this approach, which modifies only the master page (inherited by each web part page). I only need to touch one file and be done with it.

1. 
   
   In SharePoint Designer, find your master page. 
2. 
   
   Find the line :
   <asp:ContentPlaceHolder id="PlaceHolderLeftNavBar" runat="server">
3. 
   
   Directly beneath it add:
     </asp:ContentPlaceHolder>
   
4. 
   
   Find the real closing   </asp:ContentPlaceHolder>  element and remark it out like
      <!--   </asp:ContentPlaceHolder>  -->
5. 
   
   Find the line:
      <asp:ContentPlaceHolder id="PlaceHolderNavSpacer" runat="server">
6. 
   
   Directly after it add:
     </asp:ContentPlaceHolder>
7. 
   
   Find the real closing tag and remark it out
     <!--   </asp:ContentPlaceHolder>  -->
8. 
   
   Save the file

Essentially, we're letting HTML render the Quick Launch control on each page rather than let ASP manage it for us from the server. It's no longer conditional.

stsadm export/import vs Backup/Restore vs ExportWeb

There are 4 ways to Backup/Restore different sharepoint entities.

1. stsadm export/import:
• You can import single web(including all subwebs) using export method. 
• It can also export sitecollection and webapplication as well.
• It exports data using Object Model and generated new GUID for every object.
• It does not copy any information related to workflows. If import creates new lists and a workflow was associated to them, you must recreate it manually. Additionally, it does not copy workflow history and tasks lists. 
• You can run import operation on the same content database from which the export of the web was done as all the GUIDs are generated again, which you can't do in backup/restore.
• It also activates all the features.
2. ExportWeb method(sites.asmx web service Object Model):
• You should use this method if you would want to backup only single subweb without any child subwebs.

• public int ExportWeb (string jobName,
      string webUrl,
      string dataPath,
      bool includeSubwebs,
      bool includeUserSecurity,
      bool overWrite,
      int cabSize)

• 
  Parameters:
  jobName
  The file name to use for content migration packages. 
  webUrl
  The URL of the site to export.
  dataPath
  The directory location where content migration packages are placed.
  includeSubwebs
  true to include subwebs; otherwise, false.
  includeUserSecurity
  true to include site security groups and group membership information; otherwise, false. 
  overWrite
  true to overwrite the content migration package if it exists; otherwise, false. 
  cabSize
  Indicates the maximum size for the CMP (content migration package) file. 



3. Content Deployment


• Content deployment copies content from a source Microsoft Office SharePoint Server 2007 site collection to a destination site collection. The entire source site collection can be copied (full), or a subset of sites can be copied (incremental). In either case, content deployment is incremental by default, deploying only changed pages and related assets (such as images). You can also do a full deployment of all content; however, you should only run a full deployment job on an empty site collection. Also, a Quick Deploy feature supports deployment of a single page by authors.
4. Central Admin Backup/Restore(stsadm -backup/-restore):

• You can backup sitecollection,webapplication,farm using this.
• It keeps every GUIDs except one (and the most important): when you restore the file, it generates a new GUID for the site collections.
• This was done on purpose because Sites table of configuration database uses SiteID as primary key.
• This is very important because it allows you to restore the backup in the same farm in which you did the backup. You can even restore the same backup several times in the same farm, but you must always restore in a different content database, because all others GUIDs remain unchanged.
• This operation is probably more designed for SharePoint administrators that want to backup data. Because it does not do anything else than dumping the SQL content database, dministrators can be sure no data will be changed, transformed or lost.
• Any Custom solutions deployed (be sure to keep the solutions so they can be easily re-deployed).
  • Alternate access mappings
  • The Central Administrator Web Application
  • The Central Administrator content database
  • The Internet Information Services metabase

Code Access Security and SharePoint

Hi,
Don't change the trust level to Full in web.config and deploy your web parts to GAC as it is not recommended practise in most of the cases. Instead you should create CAS policy.

Example to add a CAS Policy for a sample WebPart.

Strong name the webpart and find out it's public key blob using sn -Tp assembly name.

Then in manifest.xml file for the Solution(WSP) put following entry.

<Solution
SolutionId="b172f003-d6c7-3e13-f737-a5d84a02e3e5" xmlns="http://schemas.microsoft.com/sharepoint/">

<!--Solution
created by InstallAssemblies.exe on 8/15/2006 7:03:53 PM-->

<Assemblies>

<Assembly Location="MyCustomWebParts.dll" DeploymentTarget="WebApplication">

<ClassResources>

<ClassResource Location="MyCustomWebParts.pdb"
/>

<ClassResource Location="en\two\generic.txt" />

<ClassResource Location="de\two\specific.txt" />

<ClassResource Location="header.html" />

</ClassResources>

<SafeControls>

<SafeControl Assembly="MyCustomWebParts,
version=1.0.0.0, Culture=neutral, PublicKeyToken=636addc00401d15b" Namespace="Example.CodeAccessSecurity" TypeName="*" />

</SafeControls>

</Assembly>

</Assemblies>

<DwpFiles>

<DwpFile Location="StandardAspNetWebPart.webpart" />

<DwpFile Location="HybridWSSWebPart.webpart" />

<DwpFile Location="StandardWSSWebPart.dwp" />

</DwpFiles>

<CodeAccessSecurity>

<PolicyItem>

<PermissionSet class="NamedPermissionSet"
version="1" Description="Permission
set for custom test WebParts">

<IPermission class="AspNetHostingPermission"
version="1" Level="Minimal" />

<IPermission class="SecurityPermission"
version="1" Flags="Execution" />

<IPermission class="Microsoft.SharePoint.Security.SharePointPermission,
Microsoft.SharePoint.Security, version=11.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c"
version="1" ObjectModel="True" />

<IPermission class="System.Net.WebPermission,
System, version=1.0.5000.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
version="1">

<ConnectAccess>

<URI uri="https?://.*" />

</ConnectAccess>

</IPermission>

<IPermission class="System.Security.Permissions.SecurityPermission,
mscorlib, version=1.0.5000.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
version="1" Flags="ControlThread,
UnmanagedCode" />

<IPermission class="System.Security.Permissions.EnvironmentPermission,
mscorlib, version=1.0.5000.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
version="1" Read="UserName"
/>

</PermissionSet>

<Assemblies>

<Assembly
PublicKeyBlob="hexadecimalValueTruncatedForClarity"
/>

</Assemblies>

</PolicyItem>

</CodeAccessSecurity>

</Solution>

 

Schema rules:

1. There can only be 1 CodeAccessSecurity node per Solution manifest.
2. There can be any number of PolicyItem nodes.
3. Each PolicyItem node can contain only 1 PermissionSer and Assemblies
   node.
4. Each PermissionSet node can contain any number of IPermission nodes
5. Schema of IPermission nodes are defined by the .NET framework
6. Each Assemblies node can contain any number of Assembly nodes. 
   
7. The CodeAccessSecurity/PolicyItem/Assemblies/Assembly node is different
   than the Solution/Assemblies/Assembly node.
8. The Assembly node can contain any of the following attributes:
   • Name
   • Version
   • PublicKeyBlob