Openverse contributors discussed priorities for March and April in this meeting. View the full notes here and a summary below.
Open Bug Fixes
We discussed a number of open bugs. We’ve been balancing a lot of project work with meta work (monorepo cleanup in particular) and some bugs have languished a bit.
Solution: We added assignment of these bugs to the community meeting agenda for next week. We also deprioritized some high-priority issues that went unaddressed since fall of 2022:
- #684 Tests make outbound network requests to providers for link validation and thumbnail generation
- #504 Filter sidebar is not synced correctly with page navigation causing page jittering when navigating between search and a single result
- #831 Add configuration to aggregate ingestion errors over reingestion
Current Infrastructure Issues
We discussed some issues that have recently appeared, namely:
- The bad data node on our production ES cluster in need of replacement
- The issues with the data refresh after adding 155+ million images from iNaturalist
- Ongoing work to migrate the API to ECS
Solution:
- We will immediately pause all data refreshes.
- We will mitigate the ES data node situation this week.
- We will use this ‘pause’ in the Catalog as an opportunity to run some outstanding cleanup work, DAG enables, and to plan for how to re-enable the data refreshes.
- We will re-prioritize and begin the Popularity calculation optimizations (Matview refresh) #433 project this month, with Staci as lead.
Search Relevancy Sandbox
Originally slated to begin this month, we’ve decided to defer this project to April given the infra issues previously mentioned. Madison will lead the project, and we’d like to expand the scope from the single PR of making the staging API contain a proportionate subset of images from each provider to also include investigating and provisioning a managed ElasticSearch cluster to use in staging, but ultimately migrate to in production.
Solution:
- Defer to April
- Madison will lead
- Expand scope to include a manged ES cluster for staging
Other
- We remain confident in our ability to close out the Provider DAG stability milestone this month
- We will pause design work on the Filter and blur sensitive results by term matching #377 project after getting feedback on the latest design iteration. This is to prevent additional assumptions from creeping into the work before the project is sufficiently planned.
- Regarding the consent-based desicionmaking proposal shared earlier in February, we are going to wait for a final document to be shared for final iteration and feedback before implementing. Until then, our project process contains some guidelines on communication timelines.