Algolia search index crawler must be updated every release

[jbw976]

What's Wrong?

I noticed during the v2.3 release that the Algolia crawler hasn't been indexing anything since v2.1. This may have happened while we were adjusting the config/settings for #1051, but I'm not sure. I noticed when it finished crawling and gave me an error/warning about the large number of indexes that would be dropped from the database.

I fixed it by manually adding v2.2 and v2.3 to the configuration at https://crawler.algolia.com/admin/crawlers/b1863584-45fc-4117-831f-8d68e47b2e72/configuration/edit to include the currently supported set of versions, then re-launched a crawl.

pathsToMatch: [
        "https://docs.crossplane.io/latest/**",
        "https://docs.crossplane.io/master/**",
        "https://docs.crossplane.io/v2.3/**",
        "https://docs.crossplane.io/v2.2/**",
        "https://docs.crossplane.io/v2.1/**",
        "https://docs.crossplane.io/v1.20/**",
        "https://docs.crossplane.io/knowledge-base/**",
        "https://docs.crossplane.io/contribute/**",
        "!https://docs.crossplane.io/latest/concepts/composition/",
      ],

The docs.crossplane.io site is now returning latest results from v2.3.

I'm not sure if we should just update the docs release issue template or if there's a more robust config fix on the Algolia side, but we should do something to address this or it will happen again.

/cc @haarchri

[haarchri]

we never finished the Work: #1051, we should add an promote step which Updates the paths to match

[jbw976]

yep yep makes sense @haarchri - what's left there? looks like some outstanding questions from my last review, e.g a question about the expected workflow once that PR is done, i.e. one that is very relevant to this issue here:

does the crawler config (editor link) actions.pathsToMatch also need to be updated for each release?

let me know if there's more I can help with to get that work done! 🙇‍♂️