thumbnail

Build an Enterprise Search that Helps Your Employees

Finding relevant and comprehensive information, on demand, from your entire business platform helps you get the job done faster, reducing babbling and knowledge gaps. Going for a turn-key solution (like Microsoft Search) gives you the benefit of already having pre-processing, metadata tagging, indexing, and system integration in place. 

What is Enterprise Search?

poster
Photo by Markus Winkler on Unsplash 

Enterprise Search is a piece of software which enables people from across a business or an organization to find what they need when they need it.

Failure to find relevant information at the right time leads to time waste, unused resources and untapped business potential, double work effort and manual work, and even knowledge loss. In time, poor Enterprise Search can turn into the invisible root cause for employee disengagement and churn, and for a shabby work culture. In other words, employees are expected to perform at their job, but if they cannot find what they need they cannot possibly deliver to their full potential.

Enterprise Search also comes in handy in business-critical scenarios like accreditation, audits, legal investigation, reporting, or certification, when specific and accurate records need to be produced fast or on the spot. 

What are the pillars of Enterprise Search?

poster
Photo by Tarang Dave on Unsplash 

Enterprise Search searches for data across an organization: emails, DMS or CMS, various (relational) databases, intranets, or other knowledge hubs. In general, the process unfolds as follows: 

1. Data is pulled from or pushed from multiple data sources, via connectors and crawlers. 

2. Data is pre-processed, and then indexed. The data structure is parsed and converted into a searchable structure, also known as an indexed structure. If data is tagged and semantically enriched by the entities who produce that data, the indexing is even more powerful.       

3. Data can now be queried by anyone. A search request (word, phrase, question, with or without operators) is launched, the index is queried, and the results are returned. The accuracy and relevance of the results depend on the algorithms used on the level of semantic enrichment of the content. 

4. Data is ranked, meaning that the search engine triggers the order of the results. 

Metadata and semantic enrichment 

As users, we perceive search as the process which retrieves information if we give the right input or query parameters. 

The quality of any search activity is a matter of: 

  • Metadata and semantic enrichment 
  • Data parsing and interpretation 

Additional data (or metadata) needs to be attached to data or content, so that this content is identified as the relevant answer to a query. There are several types of metadata. Here are just some of the examples: 

  • Keywords     
  • Source         
  • Author        
  • Creation date        
  • Time stamp        
  • Version         
  • Folder         
  • Business process     
  • Product 

The process by which you add metadata to your data or content is called annotation and this activity precedes any Enterprise Search implementation. Metadata tagging or assignment can be automated to a certain extent and some Enterprise Search solutions, like Microsoft Search, do provide an out-of-the box solution for the Microsoft 365 suite. However, for specialized domains and fields, organizations must enrich their data and content semantically (with tags and metadata) if they want to implement Enterprise Search successfully. 

How can you semantically enrich your data or content? 

 1. Devise a list of tags and metadata you want to apply consistently across your organization.

The list can consist of a simple enumeration of unrelated or partially related metadata, but it is best to give your metadata structure and hierarchy, which means that your list will be driven by taxonomy. To go even further, you do not need to reinvent the wheel, especially if you are working in an existing domain. Chances are that your domain already has a specialized taxonomy (also known as an ontology or an Enterprise Knowledge Graph) which you can plug and play, or customize.

2. Embed the metadata repository in your data and content management solutions.

The metadata should be available for retrieval and selection each time data or content are produced within your organization. For consistency, access should be given to the latest metadata repository version.   

3. Apply the metadata.

This should be mandatory for all staff involved in generating data or content or in annotating received data. The process can be automated or partially-automated with the right tools.       

4. Maintain the metadata.

The metadata repository needs to be maintained in a single-source which is integrated with (embedded in) all data and content-related repositories accessed by search. 

Algorithms

Algorithms are the ones which tell processes what to do when a query is launched. Some algorithms are better than others.

Whether you purchase them as part of an Enterprise Search solution, or devise them on your own, you should understand how algorithm search outputs may look like. 

Word mashup

Algorithms search a text which is not tagged in any way. Search is done based on specific words with disregard for the sequence in which words appear.   

Document tagging

Algorithms search a text based on its metadata tags or keywords. Documents are categorized, but to retrieve the best results, an exact match of the tag and of the query word must be entered by users. 

Taxonomies

Objects are arranged and classified in a more granular fashion. There is in-depth tagging and semantics captured by a taxonomy. Algorithms query the taxonomy structure, and there is no need for an exact match of the tag and of the query word to be provided by users.   

Ontologies

Ontologies are domain-specialized taxonomies. Ontologies or knowledge graphs can be purchased per domain and extended, or built from scratch. Algorithms query through a controlled vocabulary, a consistent and well-structured schema with well-defined properties. 

Algorithms are closely linked to search index calculation, which assigns a relevance score for each result. Search index calculation is a function which takes into account term frequency, document frequency, and inverse document frequency (how unusual a word or asset is in terms of occurrence). 

Monitoring and improvement

Having the data tagged and the algorithms in place does not guarantee the best Enterprise Search from day one. It is important that you monitor the results, the traffic, the retrieval speed, and the consistency and long-term relevance of your tags/properties/metadata.

For assessment and sanity purposes, test your search capabilities against expected: query volume, query response times, number of concurrent users, number of dead-end or zero-result queries, and index size.

For feedback loop purposes, test how users interact with search: number of active users, download or clicks after search results (even bookmarks and sharing), satisfaction score (if rating is implemented). 

What use cases does Enterprise Search have?

poster
Photo by Isaac Smith on Unsplash 

Business know-how

Business know-how is an ongoing process where knowledge is built and gathered collectively. As knowledge expands or changes, it is essential that all interested parties within a company have access to it. Enterprise Search enables employees to find, discover, or monitor: 

  • Specific documents or snippets of information across documents 
  • Aggregate data about customers, products and employees 
  • Latest changes on a specific topic or subject 
  • Similar patterns and multiple occurrences of an instance 
  • Divergent patterns or information 
  • Filtered data that needs to be further fed to BI tools 
  • Cross-sale opportunities based on customer profile 

Enterprise Search can integrate with dashboards. 

HR profiling

HR is tasked to attract the right candidates and to engage existing staff with the right project. Enterprise Search enables HR to: 

  • Match candidates against job descriptions 
  • Match skills against relevant projects 

Audit and compliance

Audits check how well your business observes various rules and regulations. Enterprise Search facilitates the retrieval of relevant information and documents during audits: 

  • Find specific contracts and compile reports based on chosen criteria   
  • Demonstrate the implementation of standards across the company 
  • Identify gaps or partially-compliant assets before the audit by indexing the checklist criteria 

I want Enterprise Search. Where do I start?

poster
Photo by Jukan Tateisi on Unsplash 

Good questions precede good choices. Here are the three most important steps to take before choosing or building Enterprise Search. 

1. Understand what you want to achieve with your Enterprise Search depending on the use case. 

  • Will the search engine be used internally for business continuity, staff knowledge management, research and development?   
  • Will the search engine also be extended to external, customer usage? 

2. Identify what the market has to offer and understand how much work is needed from your part or from an IT contractor to tailor the enterprise search to your needs.  

Here are a couple of options and the level of engagement needed from your side:     

  • Self-hosted and self-managed

You are responsible for building everything from scratch, and for managing it end-to-end (software and infrastructure alike). This is extremely difficult to achieve, costly, and is suitable for large corporations which have solid IT teams.   

  • Managed service from a third-party (SaaS or PaaS)

You purchase a search engine and use it, but are no longer concerned about infrastructure maintenance. You need to integrate it with your system and tailor it. 

3. Make a list of the main features you want covered by your Enterprise Search. 

3.1.  Data sources   

  • Which are the data sources that you want indexed?   
  • Can search tap into my relational DBs, DMS or CMS?

3.2.  Data pre-processing 

  • Will I be in charge of preparing the data before it gets indexed?   
  • Will I resort to a search engine which has embedded pre-processing capabilities?

Pre-processing may include data cleansing, normalization, tagging, and many others.

3.3.  Data loading 

  • Will the data be pulled from a data source (via a connector)? 
  • Will the data be pushed from a data source (via crawlers)?

3.4.  Data query algorithms

  • How does querying unfold? 
  • What algorithms does it use?   
  • How much do I need to invest in tagging, taxonomies, and ontologies? 
  • How much research do I need in order to come up with relevant search queries?

3.5.  Data query UI and NLP 

  • Is there embedded UI support or will I need to invest in building the UI?   
  • Is there multi-lingual support for queries? 

3.6 AI  

  • Is there AI which can learn from and improve search overall?