Scaling cron jobs on multisite? Here is Cavalcade, a horizontally-scalable WordPress jobs processing solution.
At the heart of every web application is a basic process: receive a request, return a response. With the right architecture, this process is able to serve everything from the smallest site all the way up to the very largest. Once sites start getting more complex, there’s quickly a need for two separate-yet-related abilities: scheduled tasks, and asynchronous processing of long-running tasks. WordPress includes the ability to do both of these through a system called wp-cron. It offers scheduled and repeating tasks (just like cron), and can be used for asynchronous processing. However, it has serious issues when running at scale, like unreliability, sequential processing, and compatibility with multisite. Replacements require complex setup processes, don’t integrate well with WordPress, or don’t scale for real production use.
To fix these problems, we built Cavalcade, a horizontally-scalable WordPress jobs processing solution. We’ve been running Cavalcade in production for almost two years, and it’s also in use on WordPress.org, so we’re confident in its stability and capability.
Limitations of wp-cron
Cavalcade originally came out of project requirements
To "tackle the problem of serving large volumes of images", Human Made has developed a scalable image service that integrates with Amazon S3.
As an enterprise web development agency, we frequently deal with high-traffic, high-bandwidth sites. We use a horizontally-scalable architecture built on Amazon’s AWS platform to ensure great performance, high availability, and low costs for our clients. This allows us to serve essentially any amount of traffic to sites without breaking a sweat. While most sites are primarily text-based, the larger size of images means that bandwidth from images can have an outsized effect on bandwidth cost and server load.
To tackle the problem of serving large volumes of images while minimising costs, we developed Tachyon, our scalable image service. Tachyon integrates with Amazon S3, and integrates with WordPress through the Tachyon plugin combined with our S3 Uploads plugin (but can also be used for non-WordPress projects).
Our First Attempts
When we initially looked at solving issues around images, we set out to solve two main issues: image regeneration, and caching. Rather than creating and storing thumbnails on upload, we wanted a dynamic system, which would allow us to easily create or change the available sizes, along with allowing complex cropping. To combine the dynamicism of this system
Really interesting news. Congratulations Mike! I might also add - thank you for WordPress!
It’s with huge excitement that I can announce that Mike Little is joining Human Made. Mike is the co-founder of the WordPress project, having forked a small blogging script, b2, with Matt Mullenweg. He has a long history with the project and brings with him unparalleled knowledge and experience of it. Mike will be joining Human Made as a WordPress Specialist; in addition to doing development on our client projects we’re exploring ways for Mike to use his extensive talents in training and outreach, where we think he can have a big impact both on Human Made and on the wider community. He’s a massive asset to the team, an old friend to many of us, and we’re honored that he will call Human Made his home. I am excited to join the Human Made family. I’ve known many of the team for quite some time and have worked with the company in the past, so I feel most welcome.
I’m looking forward to being able to work on larger projects. Something I couldn’t manage in my one-man company. I’m anticipating being able to hone my skills and take them to a new level.
Anyone who knows me knows I am passionate about free software and the GPL. I admire Human Made’s
That's the right approach. Well done team at Human Made!
Contributing to open source occurs in a range of ways; more traditionally it may involve forums, patches, documentation, translations and bug reports. But it also includes; Organising events
Speaking and volunteering at conferences
Writing about adopting open source and about contributing
QA and testing
Moderating discussions on conversation channels
Managing projects to move them along and clarify issues
Designing layouts and interfaces to improve usability
A human approach to contributing
As we reach 60+ people, we’ve been exploring more effective ways we can track the contributions we’re doing on an individual level, so we can see how we’re contributing as a whole. Not because we want to track anyone’s time, or micromanage what people are doing; but because we wanted to really understand the ways in which we were actively participating towards the health and continuity of WordPress. This conversation led us to ask questions about how we contribute as a whole; and how, looking back to the last seven years, it’s become an integral part of who we are and how we work.
We haven’t always had the same capacity to
Who wants to be the next ManageWP competitor!? :-) Or maybe MWP can scoop it up and improve/iterate on their own service with this (hint, hint Vladimir!). Regardless, it's always interesting to see businesses go up for sale because the authors couldn't dedicate the development time or get the income they expected from it.
WP Remote, our remote WordPress management web app has been helping WordPress agencies, freelancers, and site owners monitor and update their WordPress sites since we launched it back in April 2010. It’s a centrally hosted web app. You sign in and add the WordPress sites that you are responsible for managing. WP Remote then monitors those sites, tracks WordPress Core, Plugin or Theme updates that are available and handles those updates either automatically or with a single click. It additionally supports installing and managing plugins across your sites and taking backups. It keeps a log of the things that have been done and will send a daily email notifications when updates are available. It’s primarily designed to help freelancers and agencies maintain the sites they build for their clients.
WP Remote works, is used and is liked.
The people who use WP Remote like it. We regularly get feedback from people telling us they use it because it’s well designed, easy to use, and robust. The fact that it allows an unlimited number of sites to be tracked for free also helps differentiate from the competition.
It’s very well built, having been architected and developed by some of the best WordPress
Again, I can't believe no one shared this here yet, as I've seen it all over. HumanMade published a glossy 40-page white paper introducing Gutenberg. You may want to share it with clients to introduce them to the concept, though it seems to be aimed mainly at developers and advanced users.
Today Gutenberg begins as an editing tool; in the future, it will become the foundation for customising and building websites across the whole WordPress platform. A balanced look at Gutenberg's potential and wider impact, for agencies and enterprises.
How and why Gutenberg can benefit content creators, publishers, developers, and designers in your organisation
Gutenberg's potential and how to overcome the challenges of implementation and adoption
What to expect in a future with Gutenberg
Contributors to white paper
A WordPress REST API White Paper by Human Made. "...We’ve used all of this internal knowledge to produce Talking to 25% of the Web: An in-depth report and analysis on the WordPress REST API."
Human Made have found a buyer for WP Remote, their remote WordPress management web app. Read more about it here.
In 2015, we wrote about giving WP Remote a new home. In 2017 we found the perfect buyer, and now we’re delighted to announce that the future of WP Remote is in the competent hands of the brilliant team at maekit. We sat down to speak to Matt Holme, CEO and Founder of maekit, and discover more about the team behind the brand, and their plans for a brighter future for WP Remote.
What is maekit?
makeit is a complete business solution for designers and creatives, enabling them to manage all of their clients under one roof. It’s a cloud based platform that handles the business of web design projects; from payments to website deployments and updates, security and associated domain names, maekit empowers creatives and facilitates the relationship between designers and their clients.
With a proven and successful business model that enabled their Australian based web design agency, MyWork, to grow from a one-man-band into a team of 25, maekit has already gone through practical application and demonstrated its potential. When we wrote about WP Remote deserving a better home, maekit realised it was a perfect fit, and a product they could integrate directly into their new platform.
Nice report detailing everything Human Made learned while using the WP REST API for their clients, as well as what they learned while helping to develop the API.
Human Made is putting on an online conference dedicated to remote working. Event is free.
We’re thrilled to announce our latest event, Out of Office, an online conference for remote companies and professionals that will be streamed live on November 29th from 4pm – 8pm (GMT). Why?
We’ve been a distributed agency since 2012. With 36 employees across the globe, we wanted to explore questions about remote work and share what we’ve learned along the way.
On November 29th, from 4pm (GMT) we’re bringing together five professionals from leading remote companies to share their knowledge and experience, address some of the most common questions, and look deeply into the issues affecting remote workers and teams.
The entire event will be streamed live in the spiritual home of the remote worker, online. The event is free but viewer numbers are limited so register now to reserve your place and keep up-to-date with speaker and schedule announcements.
You can visit our website for more information, where we’ll soon be announcing speakers! Get in touch with us if you have any questions – we’d love to hear from you!
We’re really looking forward to seeing you there