SharePoint crawling and indexing are two processes which are used to make the content searchable. Crawling is the act where SharePoint goes through all documents on a website in order to extract both content and metadata. Indexing is the act of putting together the extracted data in a searchable database for retrieval during a search. As described by the Microsoft SharePoint, in SharePoint Online, crawls are automatically scheduled by Microsoft in a predefined period after which the content is searchable within 15 minutes to 1 hour of upload. In SharePoint Online there are three kinds of crawls; these are full crawl (crawls the entire site), Incremental (crawls only changes made since the previous crawl) and Continuous (checks change logs after every 15 minutes).
SharePoint is a fantastic tool for businesses to organize and manage their files. Thanks to the power of Microsoft Search, users have the ability to comb through whatever number of files your organization has and find exactly what they’re looking for. However, this only works if your content is properly crawled and indexed. It’s okay if you don’t know what that means – most users don’t. By the end of this guide, you should have a solid understanding of SharePoint crawling and indexing so you can take full advantage of everything SharePoint Online has to offer your business.
Content:-
The terms “crawling” and “indexing” are used interchangeably. Both refer to the organizing and cataloguing of information within your SharePoint site. While they refer to the same process, what part they play in that process differs. Crawling is the mechanism that your site uses to scan all the documents inside of it. Indexing is the process of your site sorting and integrating this information into its search database. Once a document has been indexed and crawled, it is added to your site’s search index, making it eligible to show when a user performs a search related to the content.
Crawling and indexing is the two-step procedure which underlies the work of the SharePoint search function. While crawling, the SharePoint search engine analyzes libraries, lists, pages and other sources of data in order to extract information in the form of text, metadata and other properties of documents. The result of the indexing procedure is a search index, which is a database where keywords and metadata values are linked with corresponding documents. As stated in the Microsoft's SharePoint index description, the index is the base for all searches - when the user enters the keyword, SharePoint performs search in the index and not directly in the documents. That is the reason why the data should be crawled and indexed in order to appear in search results. In case of SharePoint Online, both procedures are executed automatically.
SharePoint sites can perform three different kinds of crawling.
Full Crawl: A full crawl is when the crawler sifts through the content and metadata for your whole site. The time this takes varies depending on the number of files your business is storing.
Incremental Crawl: An incremental crawl is when the crawler only sifts through items created or updated since the last crawl.
Continuous Crawl: A continuous crawl is when a crawler checks the change logs on your sites regularly (every 15 minutes is the default).
For the most part, your site will crawl once every 4-8 hours, although this will depend on what version of SharePoint you have in use at your company.
There are three types of crawl for SharePoint Online, and all serve a particular function in making sure your search index stays up-to-date. The full crawl crawls everything within your content source: documents, pages, list items, and their metadata, no matter whether there has been a change or not.
Full crawls are costly and are usually done following any major changes to the configuration, such as schema changes or additions of new managed properties. An incremental crawl crawls only the items that have changed, been deleted, or added since the last crawl. This is the default crawl schedule which happens automatically by Microsoft's managed schedule in SharePoint Online. The continuous crawl complements the incremental crawls, checking change logs every 15 minutes and making sure high priority changes get indexed promptly.
As mentioned in Microsoft's documentation about index management, in SharePoint Online the crawl schedule is managed entirely by Microsoft and is not able to be set manually, although a re-index can be started manually.
As stated previously, the search index contains all the searchable content within your site. What documents your search content retrieves is determined by your SharePoint site’s search schema. The search schema is made up of crawled properties, crawled property categories, the crawled to managed property mapping, and the managed property setting.
A crawled property is the content and metadata that the crawler extracts from an item. This can include the author, title, or subject. To include this information in the search index, you must map the crawled properties to managed properties in your SharePoint site. Managed properties are the attributes that determine how your content shows in search results. If you do not map a crawled property to a managed property it will not be entered in the search index.
Your organization’s search schema controls what users can search, how they search, and how the results are presented on your intranet. It’s important to make sure that your search schema is tooled to match your business’s needs. Take the time to invest plenty of resources into managing your search schema to make sure your search index is providing users what they are looking for in a timely, convenient manner.
The SharePoint search index is the database in which information about all crawled content is stored in a searchable form. This means that the database contains text and metadata values extracted from documents and other properties to make search queries work fast. In Microsoft's index management documentation, it is stated that the search index in SharePoint Online is maintained at the tenant level and is common for all sites in the tenant. This means that the index contains the full text of documents (to enable searching for keywords) and metadata values (to enable filtering or refinement of search results). When a user makes a search query in SharePoint, the query is processed in the index according to certain criteria (such as frequency of usage of keywords and managed property values) and users' permission levels. If there is any content in SharePoint that has not been crawled or for which crawling was unsuccessful, then this content won't be found in any search query results, no matter how high the users' permission level is.
It is crucial for anyone implementing enterprise-level search in SharePoint to be aware of the differences between crawled and managed properties. Crawled property is a type of metadata that gets extracted from documents or list items during the crawling process. For instance, the author of the document, title of the document, last modified date, size of the document or custom metadata columns added to a document library.
The crawled properties are discovered automatically by the crawler and are not available for searching or filtering by end-users. Managed property is a named, indexed property that appears in the search results, refinements and in the query filters. In order to be able to use the crawled property in the search, it needs to be mapped to a managed property in SharePoint search schema.
There is official Microsoft Learn documentation about search schema that will guide you through creating new managed properties and mapping crawled properties. If your organization has complicated requirements regarding its metadata – such as using contract management system and having custom columns like contract type, vendor name or expiration date – then it is worth spending some time configuring the search schema.
Unindexed content cannot be searched. To have newly uploaded file or document indexed, you should either wait until the next crawl takes place or request re-indexing. The procedure to do this will depend on the version of SharePoint that your company uses. The crawl scheduling of SharePoint Online is automated and cannot be modified. It is managed by Microsoft and generally takes from 15 minutes to an hour from uploading the file/document until it becomes available for searching. In turn, the crawl scheduling of SharePoint On-Premises can be altered and manual re-indexing of the SharePoint site is possible.
There are some things you should remember if your company uses SharePoint on-Premises regarding the process of manual re-indexing of the SharePoint site. First, if you add or update managed property list or library, make sure that you change the search schema, but not the site itself. Modification of the site will create great trouble for your business. Second, make sure that you choose the right crawling type for you. Full indexing of your SharePoint site may take much time and slow down the searching process greatly.
The action of re-indexing a site or a SharePoint list means that all the content in it will need to be re-crawled in the following indexing cycle, irrespective of the change status of individual items. There are some definite occasions when this needs to happen.
The first one is when changes are made to the search schema - in case new managed properties have been created, in case property mappings have been altered or search settings have been changed, the existing content will need to be re-indexed to reflect those changes. The second situation is when there is a large content migration into SharePoint from somewhere else. The third scenario is when there is a problem with the availability of the content in the search results and no amount of troubleshooting helps. And finally, after a restoration of the site or library from a backup. In accordance with Microsoft's index management information, re-indexing does not remove the current index but just marks all the items for re-crawling and creates new index entries during the crawling process. It might take a few hours in the case of larger sites.
If you need to manually re-index your site, here are the steps you need to follow:
A business’s success hinges on its users’ abilities to find the files they need in a timely manner. At Dock 365, we've seen the impact that quick content retrieval can have on an organization’s effectiveness. That’s why we designed our Document Management System to efficiently sort and manage your documents and files.
Book a free demo today to see how effective our intuitive, user-friendly system is!