Earlier this year, representatives from the themes team and meta team had a call together with Matt and Josepha. This set a new goal and action points for improving the theme submission process.
Meeting notes February 24, 2021
Next steps on themes and reviews
Overarching goal
Increase the number of installed themes that originate from the WordPress.org theme directory.
Action points
- Reduce requirements.
- Improve automated tools.
- Move the manual review step to after a theme is added to the theme directory.
- Add user facing flags for theme quality based on the result of the automated checks.
- Add a new algorithm for the theme directory that prioritizes themes based on the flags.
- A local development environment with the theme directory to make it easier to contribute.
- Make it easier to submit themes.
Summary of what has been achieved so far
During quarter three (Q3), the focus has been on automated tools and reducing requirements.
As of October 4, the review process still includes a manual step before a theme is added to the theme directory. Before the manual review can be moved, an automated license check must be implemented, and the security checks must be analyzed.
The requirements have been reduced to a list of 13 items. This list will be reduced further when the order of the review process is updated.
- Theme Review Action has been implemented on the themes Trac and is used to present results from e2e and integration tests of the theme.
- Theme Check plugin (which is also used on the themes Trac) has been updated to reduce the requirements and improve error messages.
- A local environment for the theme directory is available at https://github.com/WordPress/theme-directory-env
- A plan for allowing themes to be submitted via SVN has been published.
- Work on the user-facing badges or flags, and the new algorithm has not started.
Further reducing requirements and improving automated tools
One goal, with reducing the requirements and improve the automated tools, is to reach the point where authors can safely add themes to the directory without manual review.
The current review process
In short:
- A logged-in user submits a zip file.
- Automated review.
- Manual review.
- The theme is added to the theme directory manually.
What we aim to change the process to
- A logged-in user submits a zip file or uploads a theme via SVN.
- Automated review.
- The theme is added to the directory automatically.
- Results of the remaining automated tests are presented as flags or badges.
- Manual review after user reports, spot-checking.
The automated review at step 2 is proposed to make sure that:
- The theme is secure.
- The theme is GPL compatible.
- The WordPress trademark is respected.
- The theme does not break the theme directory or theme previewer.
Requirements
Because engagement is understandably low because of covid, I decided to list the easy, or if you prefer, low conflict and non-controversial requirements first, publish a proposal from that and then iterate.
To decide which requirements to remove, I used the list of guard rails and looked at the impact of removing the requirement on the theme directory itself and theme quality.
I used a public GitHub repository to create an overview of the requirements and to present suggestions of requirements that could be removed. There was no engagement besides those directly involved in the project.
A proposal for reduced requirements and request for feedback was published on the Themes team Make blog on July 20. The team had a zoom call with theme authors to discuss the changes.
With the result of the request for feedback, two more requirements were reduced.
@dd32 created a new Slack channel called #themereview-firehose to help the themes team representatives keep track of which requirements are preventing themes from being uploaded. This type of statistics was not available before.
Automated tools
There are two main tools:
- Theme Check plugin, which does basic static code review
- Theme Review Action, which creates a WordPress environment and runs tests on the installed theme including the front end
Theme Review Action uses Theme Check, and results from both tools are presented to the theme author and reviewer.
GitHub was used to document what is actually checked by the automated tools, compared to what was listed on the theme requirements page on WordPress.org.
Besides the work on updating and creating new checks and tests, time was put into removing discrepancies.
@dufresnesteven created an improved table that lists the requirements in different categories. This helped us gain a better overview of the requirement as they were iterated on. We are using the table to list which tool is used to check which requirement.
Next steps
I would like to add that I expect the requirements to be reduced further if the themes team can find a way to engage more people in the conversation.
To help theme authors adjust to the new less restrictive requirements, I propose that the themes team publishes more in-depth articles to clarify the changes.
My analysis is that three more steps are needed to reach the point where authors can safely add themes to the directory without manual review:
- Determine if the security checks are enough, and if not, learn what still needs to be added.
- GPL license confirmation
- The PHP error check in Theme Review Action must prevent theme upload, to prevent the theme previewer from breaking.