Welcome to ManageWP.org

Register to share, discuss and vote for the best WordPress stories every day, find new ideas and inspiration for your business and network with other members of the WordPress community. Join the #1 WordPress news community!


Automated compatibility tester for WordPress.org plugins

Apr. 4, 2016

Here it is again, the dreaded email announcing new WordPress version and the need to manually review and up the compatibility version on all 28 plugins I contributed. It is a battle that I can not possibly win long term.

Unless.. we can do something about it. What would it take for the WP plugin repository to have an automated compatibility suite of tests, like Mozilla Firefox for example has been utilizing for years? It seems that spinning an instance of WordPress, running the plugin in question in debug mode and logging output would be fairly straightforward to do with current tools available to devops. If someone in the community developed such a thing, what's the path for it to be adopted to wordpress.org repo?

And here are the two emails that I get from WordPress.org and Mozilla, it is easy to see which one I like much better.

WordPress.org email

Hello, Vladimir Prelovac!

WordPress 4.5 is scheduled to be released on April 12. Are your plugins ready?

After testing your plugins and ensuring compatibility, it only takes a few moments to change the readme "Tested up to:" value to 4.5. This information provides peace of mind to users and helps encourage them to update to the latest version.

Here are the current "tested" values for each of your plugins:

* wordpress.org/plugins/broken-link-checker/ (tested up to 4.4.1)
* wordpress.org/plugins/category-search/ (tested up to 4.2)
* wordpress.org/plugins/cleveritics-for-wordpress/ (tested up to 4.2)
* wordpress.org/plugins/footer-javascript/ (tested up to 4.2)
* wordpress.org/plugins/insights/ (tested up to 4.2)
* wordpress.org/plugins/live-blogroll/ (tested up to 4.2)
* wordpress.org/plugins/navbar/ (tested up to 3.3)
* wordpress.org/plugins/no-frames/ (tested up to 4.2)
* wordpress.org/plugins/plimus-for-wordpress/ (tested up to 4.2)
* wordpress.org/plugins/plugin-central/ (tested up to 4.3.1)
* wordpress.org/plugins/pretty-theme-files/ (tested up to 4.2)
* wordpress.org/plugins/seo-automatic-links/ (tested up to 4.1)
* wordpress.org/plugins/seo-image/ (tested up to 4.1)
* wordpress.org/plugins/seo-super-comments/ (tested up to 4.2)
* wordpress.org/plugins/shellshock-check/ (tested up to 4.3)
* wordpress.org/plugins/show-me-options/ (tested up to 4.2)
* wordpress.org/plugins/smart-youtube/ (tested up to 4.1)
* wordpress.org/plugins/snazzy-archives/ (tested up to 4.2)
* wordpress.org/plugins/theme-test-drive/ (tested up to 4.3.1)
* wordpress.org/plugins/what-should-we-write-about-next/ (tested up to 4.2)
* wordpress.org/plugins/worker/ (tested up to 4.4)
* wordpress.org/plugins/wp-digg-this/ (tested up to 4.2)
* wordpress.org/plugins/wp-figlet/ (tested up to 4.2)
* wordpress.org/plugins/wp-nofollow-categories/ (tested up to 4.2)
* wordpress.org/plugins/wp-quick-deploy/ (tested up to 4.2)
* wordpress.org/plugins/wp-wall/ (tested up to 4.2)
* wordpress.org/plugins/yahoo-news-feed/ (tested up to 2.9)

For each plugin that is compatible, you don't need to release a new version -- just change the stable version's readme value.

Looking to get more familiar with 4.5? Check out this roundup post on the core development blog: make.wordpress.org/core/2016/03/30/wordpress-4-5-field-guide/

Thank you for all you do for the WordPress community, and we hope you will enjoy 4.5 as much as we do.

WordPress core contributors

Mozilla add-on message

Dear add-on author,

Good news! Our automated tests did not detect any compatibility issues with your add-on and Firefox 46.*. We've updated your add-on's compatibility to work with Firefox 46.* so that our beta and release users can begin using your add-on.

We encourage you to view the results of the compatibility test, as some compatibility issues may have been detected but without enough certainty to declare the add-on incompatible:

• SEO Doctor:

This compatibility bump is server-side and we did not modify your add-on package in any way.

20 votes   Flag
Donna Cavalier

Have you considered putting some of those up for adoption? wordpress.org/plugins/tags/adopt-me

Maybe someone else would be happy to give them some attention, and have the time for things like this. Not that I disagree with the concept of making the process easier. I'm all for that. But I know from experience that some projects would be better off in someone else's hands when I don't have time to devote to them.

M Asif Rahman Ⓦ

Not each project needs continuous development. Some are working fine as they are, just need to test if its fine, and list as compatible. And I don't think Vlad wants to give up his plugin up for adoption, he loves all of those!

Vladimir Prelovac

My problem would only go away if I put all of them for adoption :) While this may solve the problem for someone having just one or two plugins, I am more for a long-term solution that is within a realm of development and that would instantly help all developers maintain their plugins.

David McCan

Yup, it is easy to see which email you prefer.

How would that work? All I can think of off the bat is that you copy the plugin into a fresh copy of the release in question and activate it ... then check for errors. Some errors appear on use, so that would only catch the first wave. How could you automate a test beyond that? There may be some search in the plugin code that happens when you first submit it, and I guess that could be run again.

M Asif Rahman Ⓦ

Remember this service? Details here - addendio.com/announcing-wordpress-1-click-demo-links/
This could be good tool to get started with.

Kyle Maurer

Probably the most appropriate place to voice such a proposal would be in WordPress Slack in the #meta channel. After getting some feedback from folks there in real time, the next step would be to open a ticket here: meta.trac.wordpress.org/query?status=accepted&status=assigned&status=new&status=reopened&status=reviewing&component=Plugin+Directory&col=id&col=summary&col=component&col=status&col=owner&col=type&col=priority&col=keywords&order=priority

That's where it would get official acknowledgement and support if it were to actually be created. Cool idea though!

Luca Fracassi

Luca here, the guy from Addendio.com

I did not think of Addendio that way, but I share the pain of updating the Readme.txt like everyone else.

question is: how would you define a automated generic test for each plugin?

a simple success/fail upon installation of the plugin with the new version of WP?

Vladimir, can you share a screenshot of what Mozilla shows in their test?

Vladimir Prelovac

There is not much to see here, as all tests passed:


I would imagine launching a WordPress instance loaded with 25 most popular plugins on the repo on minimum supported versions of PHP, MySQL plus the plugin in question, enabling WordPress debug mode and logging any PHP error would be enough for an automated compatibility test.

What most people fear is their site breaking after they update the plugin and this test would effectively help with that.

Luca Fracassi

thanks Vladimir, make sense...