Basic Full-text Search

[0xDing]

Problem

Articulate the problem that this piece of work addresses

We need to support basic full text search.

Why is this the right time to address this problem?

We should prioritize this problem to improve the basic usage experience.

Time Budget

1 weeks

Solution

• PostgreSQL full-text search
• algolia-style ui

Out of Bounds

Support for searching plain text content only.

Rabbitholes

• In the future it will also be possible to execute commands in the search box, similar to spotlight search.
• Shortcuts need to be considered

[stackia]

owner @Darmody

[Darmody]

Global Shortcuts

First, we need to introduce global shortcuts to support invoke search popover by hotkeys. react-hotkeys-hook provides an easy way to define hotkeys via react hooks. We can create a hotkeys hook for the web app.

function useGlobalShortcuts() {
  useHotkeys('mod+p', () => // show up search popover)
}

Search and Filter

• Allows searching by typing text content.
• Allows filtering by a group of filters.
• There are various kinds of filters.

To support text searching and conditions filters, the design system should have a FilterInput component that can be used directly.

Filter types

// Filters by kinds
type FilterKind = 'document' | 'media' | 'spreadsheet' | 'command' | 'and so on...'

// Filters by the range of creating date
type FilterCreatedAt = [Date, Date]

// Filters by the folder name
type FilterFolder = string

// Filters by the author id
type FilterAuthor = number

Search popover

• Search popover contains three parts: searching input, filter, and search results.
• When there are too many search results, the panel can be scrolled for more content.
• When searching popover popup, the user can select result items via mouse event or keydown event.

There are two components FilterInput and SearchingResults at the top level of the SearchPopover.

function SearchPopover: FC = () => {
  const [searchContent, filters] = useSearchingContent()
  const [activeIndex] = useActiveIndex()
  const searchingResults = useSearchResults(searchingContent, filters)

  return <Popover>
    <FilterInput searchContent={searchContent} filters={filters} />
    <SearchingResults activeIndex={activeIndex} searchingResults={...searchingResults} />
  </Popover>
}

FilterInput

FilterInput should be imported from @mashcard/design-system.

SearchingResults

SearchingResults contains the searching results that come from FilterInput.

interface SearchingResultsProps {
  // An array of searching result items
  searchResultItems: Array<SearchResultItem>
  // Active item index
  activeIndex: number | undefined
}

Showing default results or recently used results if no search content or filters are offered.

function useRecentlyResults(searchResults) {
  useEffect(() => {
    // Set recently used results to local storage
  }, [searchResults])
  return useMemo(() => // Get items from local storage, [])
}

function useSearchResults() {
  const searchResults = useState()
  const recentlyResults = useRecentlyResults(searchResults)

  // Handle searching

  return useMemo(() => searchResults.length > 0 ? searchResults : recentlyResults, [searchResults, recentlyResults])
}

The order of results should base on the relevance property of SearchResultItem.

The container showing the results array should have a max-height CSS property set and can be scrolled when content is overflowed.

const SearchingResultsContainer = styled('div', {
  maxHeight: 'a fixed max height or 100% if popover component can detect the suitable height for the viewport.'
  overflow scroll
})

We should add keydown listeners at the top of SearchingResults for selecting items by key pressing, and it also should scroll the item into view when it is overflowed.

const [index, setIndex] = useActiveIndex()

function onKeydown(event) {
  switch (event.key) {
    case 'ArrowUp':  {
      // set the active index
       setIndex(Math.max(0, Math.min(index, itemLength - 1))
      // 2. Scroll item into view. element.scrollIntoView().
    }
  }
}

Active selection should be reset when search content changed. NOTICE: it is not related to search results but search content, because search results may be changed in an async way.

function useActiveIndex(searchContent: string | undefined, filters: Filter[]  | undefined) {
 const [index, setIndex] = useState<number>(undefined)

  useEffect(() => {
   setIndex(undefined)
 }, [searchContent, filters])

 return [index, setIndex]
}

Search Result

• Searching will show various kinds of results, such as command, document, formula, spreadsheet, media, and so on. Hence a base interface of searching results is required.
• Result of searching includes brief of the item combine a full preview of it.
• Showing the preview when the result item is active (hover).

interface SearchResultItem {
  // The represent what kind of the result is.
  kind: 'command' | 'document' | 'and more...'
  // The title of the search result
  title: string
  // The description of the search result
  description?: string
  // The icon of the search result
  icon?: string
  // The preview of search result
  preview?: ReactNode
  // The action when the item is selected.
  onSelect: Function
  // The relevance of the result and search/filter content. It is useful in showing items in the result list.
  relevance: number
}

interface SearchResultCommandItem extends SearchResultItem {
  // The HTML content of the API doc of the command.
  APIDoc: string
}
// Import the styles of the API doc, a top-level className, and styling the basic markdown structure like `h1, h2, h3, a, p ...`.

interface SearchResultDocumentItem extends SearchResultItem {
 // The id of the document 
 documentId: string
 // The selection bookmarks of the matched content inside the document.
 selectionBookmarks: SelectionBookMark[]
}

interface SearchResultFormulaItem extends SearchResultItem {
 // The id of the document 
 documentId: string
 // The id of the formula
 formulaId: string
 // The selection bookmarks of the matched content inside the document.
 selectionBookmarks: SelectionBookMark[]
}

interface SearchResultSpreadsheetItem extends SearchResultItem {
 // The id of the document 
 documentId: string
 // The id of the spreadsheet
 spreadsheetId: string
 // The selection bookmarks of the matched content inside the document.
 selectionBookmarks: SelectionBookMark[]
}

interface SearchResultMediaItem extends SearchResultItem {
 // The id of the document 
 documentId: string
 // The content type of media
 contentType: string
 // The source link of the media
 src: string
 // The selection bookmark of the matched content inside the document.
 selectionBookmark: SelectionBookMark
}

Highlight the matched content

highlight the keywords inside the description.

const description = description.replace(keyword, `<mark>${keyword}</mark>`)

highlight the keywords inside the preview

See below.

How Search Work

Due to various kinds of search results, there must be multiple ways to search. Hence we need a Search Workflow.

Search Workflow

• A workflow returns the results of searching in sync or async way.
• The result shows up immediately if it is a sync workflow.
• The result is attached when an async workflow is ready.

interface SearchWorkflow {
  // Name of the current searching workflow.
  name: string
  // A function for searching. Return can be a promise.
  onSearch: (textContent: string | undefined, filters: Filter[]) => SearchResultItem[] | Promise<SearchResultMediaItem[]>
}

function useSearchResults(textContent: string | undefined, filters: Filter[]): SearchResultItem[] {
  const [results, setResults] = useState([])    

  useEffect(() => {
   // Should add debounce for searching changing.
   debounce(() => {
      // Calls every workflow.
      workflows.forEach(workflow => {
        const results = workflow.onSearch(textContent, filters)
        // Async case
        if (result is Promise) {
          // 1. Awaiting result, and it must be cancelable.
          // 2. Push results into the state.
        // Sync case
        } else {
           setResults(currentResults => [...currentResults, ...results])
        }
      })
    })

    return () => {
      // Clear the effects from the last dependencies. Like async result awaiting.
    }
  }, [textContent, filters])

  return results
}

Command Search

Command search is a sync search workflow. we can add a commands list in the web app and search for commands by comparing command names and search content.

const CommandSearchWorkflow: SearchWorkflow = {
  name: 'command',
 onSearch(textContent) {
   return commands.filter(command => command.toLowerCase().includes(textContent?.toLowerCase()))
  }
}

Full-text Search

Full-text search is an async search workflow. And the highlights of keywords should be marked on the client-side.

1. Calls SearchBlocks GraphQL query to get search results from the server.
2. Creates the preview editor for each result and highlights the keywords.
3. See more in Basic Full-text Search #393 (reply in thread)

const FullTextSearchWorkflow: SearchWorkflow = {
  name: 'fullText',
 onSearch(textContent, filters) {
    const blocks = await querySearchBlocks(textContent, filters)
    return blocks.map(block => ({
      // Map to SearchResultItem
      onSelect: () => // navigate to document page
      preview: () => {
        // 1. Render the preview document 
        // 2. Highlight the keyword
        doc.descendants(node => {
          if (node matches the keyword) {
            // Add highlight mark to node
          }
        })
      }
    }))
  }
}

Ideas

The folders in the left sidebar can be right-clicked, and a search option will show up in the context menu. The option will pop up the search popover and filter the search results in that folder by default.

[aligo]

Server-side Schema

title and content fields in blocks table (future objects table) will be indexed with gin_trgm_ops type for performing full text search, the weight of title and content will be different, for document type blocks, content will be used to store document content in plain text.

GraphQL interface

query SearchBlocks($query: String, $filters: Filter[], $page: Int, $per_page: Int) {
  searchBlocks(input: $input) {
    blocks {
      id
      type
      title
      content
      rank
    }
    totalPages
  }
}

Document Preview

1. useDocumentBlockQuery to load the data of the document id
2. useDocSyncProvider to get the document provider (editable should be set to false, turn off persistent document synchronization)
3. pass provider above to the prosemirror editor to render the document content preview, then highlight the keywords