Indexing Engine Overview

Lucy’s indexing engine uses microservices built on NodeJS to create connectors to different integrations. This allows a scalable, secure, maintainable connection to be established with a wide variety of sources and data formats. A source connector is made which scans (for new sources) and observes changes (for existing sources) and queues content to be indexed.

These source connectors are highly customizable and typically customized to each source based on the source’s available APIs, update strategy, and metadata requirements. Updates can come in the form of (or any combination of the mechanisms below):

• Polling or checks on a regular interval.
• Webhook push or checks when the source says something new came in.
• Delta scans or checks a subset of files matches a criterion.

Figure 1 – Generic Indexing Overview.

This associated integration documentation covers several integration examples which can be used to share document, facilitate rich, embeddable dashboard integrations, or render RSS feeds, among others:

• API requirements for successful integration and user experience.
• Lucy’s standard webhook configuration (customization required for dashboards).
• Sample API endpoint configured by content provider to embed dashboards.
• Supported RSS feeds. 

This documentation also provides an overview of the recommended answer user interface to provide the best experience for users after they’ve asked a question.