I made a lot of mistakes going from developing closed-source to open-source software, and want to share the unspoken rules I discovered in the process.
This post is intended for readers with programming experience (especially with WordPress), but who are new to contributing to open source projects.
My First Pull Request Disaster
Customizing BuddyPress Docs for a Client
Six years ago I was working at a company in Victoria BC Canada called The Number. We were creating a website for a government client who wanted to have a tool for its members to be able to share documents. The WordPress plugin BuddyPress seemed like a good fit because it would add the social network the client wanted, and the plugin BuddyPress Docs would allow the members to collaborate on documents.
The one feature BuddyPress Docs lacked was the ability to upload file attachments – by itself, it only supported shared documents like Google Docs, not sharing PDFs or spreadsheets or images. So we decided to customize BuddyPress Docs to add the feature our client wanted.
This illustrates a great strength of open-source software, including WordPress: it’s not only free, but it’s infinitely customizable.
So, I downloaded a copy of the plugin from the WordPress plugin repository, customzied it and found it worked fine. As an afterthought, I thought we’d “give back” to the community by submitting my changes to the master copy of the project.
The Pull Request Was Missing a Few Things…
BuddyPress Docs, like many WordPress plugins and themes, in addition to being on the WordPress plugin repository, is also hosted on GitHub. It turns out, GitHub is actually where most of the development happens, and the WordPress plugin repository is just where the finished product goes.This was the first time I’d ever attempted to create a pull request, and it took me a while to figure everything out. I thought I could somehow create a patch file, which contains all the changes, and upload it to the GitHubI created the pull request (a.k.a. patch) and sent it off. I thought the notes on the commits (a.k.a. changes) were self-explanatory, so I actually didn’t even bother giving a title to the pull request, or a description. It was the equivalent of dropping a digital baby off at the doorstep, ringing the doorbell, and running away.
So in order to get my code into the master copy of BuddyPress, I’d need to create a pull request. I’d never create a pull request before, and so it took a bit of getting used to. I incorrectly assumed I’d just use Git, the version control system GitHub is based on, to create a patch file that summarized my changes, and upload it somewhere on GitHub. I was wrong (it turns out, however, that’s basically what you do if submitting code to WordPress core on Trac). On GitHub, what you actually do is: fork the project on GitHub, thus creating your own copy of the project, then make the changes to that fork, and then use GitHub’s interface to request that the project maintainer pull the changes from your fork into their master copy (hence the term “pull request”, literally a “request to pull changes in”). Well, I was so engrossed in figured out that out, that I made a bunch of mistakes.
Because I saw my commit notes appeared in the pull request (that is, the labelled groups of changes), that there was no need to give a title to the pull request or any description. (Actually, I nameed the pull request “1.1.x”, what I thougth the plugin’s next version number would be). What I did was the equivalent online equivalent of dropping a baby off at the doorstep, ringing the doorbell, and running away.
Boone was surprisingly grateful for the submission and said he’d take a look but raised some concerns about security and future maintenance. I’m not sure why, but I said absolutely nothing in reply. I totally ignored it. Maybe I thought he was ungrateful, that he should just accept my changes as-is. Maybe I thought “This adds a great feature. It works. Why don’t you just accept it?”
Here is 3-minute video that Boone and I created recently, dramatizing our earlier encounter:
Pull Request Post Mortem
My patch made no progress for months (because I didn’t understand the issues Boone pointed out, nor did I ask him to explain). Nearly a year later, Boone closed the pull request, saying he had basically added the same feature, but in a way that didn’t have the problems he had pointed out. He said my pull request gave him some ideas how to do it, but it was rejected.
Importantly, our client was left in a really bad situation. By customizing the plugin, if they ever updated their copy of the plugin, the customizations would be lost. That’s because when WordPress updates a plugin, it completely removes the old version of the plugin (in this case, the version that had my customizations) and replaces it with the new version from WordPress.org (which would NOT have my customizations). So essentially, they were stuck forever using the same version of BuddyPress Docs. That’s inconvenient if it gets other features added later, which they won’t be able to get; it’s disastrous if the security vulnerabilities are later discovered in it, and they are unable to update to fix them.
Also, and I had grossly misunderstood that submitting code to an open source project used by thousands is much more involved than just making the changes to a private project seen by no one but yourself.
I didn’t realize it, but there were certain unspoken rules about contributing to open source software that I broke, which we’re going to dive into. In order to understand why they’re important than, it helps to realize how open and closed source software differ.
What’s The Big Difference?
Let’s Compare a Few Facts
There is a lot more to discuss and people with whom to discuss an open source project, as opposed to a one-off customization.
Let’s compare the client’s website I was building to BuddyPress Docs, and just for giggles to WordPress, too.
|Client Site||BuddyPress Docs||WordPress|
|Sites Using It||1||8,000||23% of 1,200,000,000 = 276,000,000|
|PHP versions supported||5.5.9||5.2.5-7.1.x||5.2.5-7.1.x|
|Compatible Plugins||about 10||52,000||52,000|
|Hackers||A few?||A few more?||All of Them!!!|
So the code I submitted worked fine on our customer’s server, using just their version of PHP, with the plugins they had active. But that in no way meant it would be ok on all 8,000 sites running BuddyPress Docs, on all the versions of PHP between 5.2.5 and 7.1, with all 52,000+ other plugins on WordPress.org (not to mention unpublished plugins and customizations others may have made). Also, there were 40 other contributors who might want to discuss it, each with their own schedules, experience and styles, and mood-swings. And lastly, although a security bug on our client’s website would probably go unnoticed, it would be much more likely to be exploited if it became part of master copy (and hackers would be able to inspect the code themselves).
So when contributing code to open source, there is a lot more to consider and a lot more people with whom to discuss it.
And if you want to contribute code to a big project like WordPress, all those issues a multiplied exponentially.
Should You Take the Plunge into Open Source Software?
Am I saying you shouldn’t try to contribute to open source software? Yes; if you expect it to be as quick-n-dirty as building software in isolation. But I really hope you do venture out of your coding cocoon because there is a lot to be gained. Open source contributions are great for resume-building, you’ll support the software your livelihood depends on, and you’ll basically receive mentoring from world-class professionals you’d never get a chance to even interact with otherwise.
So I honestly believe it’s worth it, but there are a few rules I’ve learnd I’d like to share.
Rule 1: Justify Your Changes
My Hand-Made Sweater Analogy
When submitting your patch or creating a pull request, you need to show to the maintainer that it’s worth the costs. I thought my patch was a beautiful hand-made masterpiece, like a piece of hand-knitted clothing, but it was actually an ugly sweater you not only don’t want to be seen in, but it may have lice and other bugs.
And even if you just store your sweater in your cupboard, the bugs may creep from it into everything else you wear. (I think the analogy to software bugs is obvious here)
Pitfalls of Patches
Project maintainers need to be very careful about what changes they accept from strangers. When deciding whether to accept your patch, they need to consider:
- does it add any vulnerabilities? (could users get hacked because of this?)
- does it add any other bugs?
- is it compatible with all the supported versions of PHP, and MySQL, and web servers, etc?
- will it conflict with other plugins and customizations?
- does it fit into their feature roadmap? (and, usually, it will make adding other new features harder)
- how will it affect performance?
- will most users use it?
- will most users like it?
- is it worth the extra work to maintain it? (because all that code may need to be rewritten the next time-related software is updated)
Although Boone appreciated my contribution, it wasn’t so much of a gift as an obligation: the obligation to evaluate it, and if accepted, the obligation to forevermore maintain it himself. Someone on twitter once said “All code is technical debt”, meaning even clean, well-written code will require work to maintain. So every code contribution adds to their workload.
An Improved Pull Request
So if you want your pull request to be accepted, the project maintainer needs to be convinced it’s worth the risk. These are things to mention:
- describe the feature (a high-level overview, they can look at the code for details)
- your use-case (why you find it useful for your project)
- other use-cases (other applications you’ve heard of or can think of)
- prior art (other projects with similar features and why it’s been good for them)
- why this implementation (why not do it another way)
What justification did I give to Boone that my feature should be accepted? Absolutely none. Did I at least tell him why I did it for my own project? Nope. Did I convince him the benefits of this feature outweighed the potential maintenance costs, bugs, and vulnerabilities? Absolutely not.
If I were to do it again, I would have added a title and description, something like this:
Rule 2: Be Willing to Discuss and Revise
My Pull Request’s Discussion
Even though I gave no justification for why the feature was worth it, Boone still followed-up with a bit of a discussion about it. To which I made absolutely no reply.
I should have realized that, of course, he’d have questions about it. If he accepted it, he was the one who was going to maintain it, answer questions about bugs it might have, and apologize to users if it introduced a security hole that made their sites get hacked. Not me, I was probably going to disappear. He needed to firmly know there was no problems with it; all his questions needed to be satisfied.
There are a million aspects to consider, and it’s unlikely my first-attempt would have satisfied them all (actually, it’s unlikely until it’s released and people start using it). Even regular contributors’ code contributions often require a lot of discussion and revisions.
What Does Silence Mean?
What did I, or anyone, communicate with silence?
- “Uh- could you repeat that? I wasn’t listening”
- “I understand so little of what you just said, I don’t know what to even ask. Please explain more.”
- Nobody’s here. Try again later, or try somewhere else.
- “I’m thinking or checking with others, please wait”
- “I thought someone else would reply”
- “I’m too busy right now”
- “I’m really tired of this and want it to just end”
When someone doesn’t reply to us (be it over text message or pull request discussion), we damage our own well-being by usually assuming one of the latter reasons; and yet often when we don’t reply it’s because of one of the former reasons. Everyone’s happier when you assume the best and try to give replies, even if it’s “I don’t understand” or “Please wait”.
A More Typical Patch’s Discussion
I’ve been following a WordPress core ticket 38645, done by one of the biggest (and most memorable) names in WordPress, John James Jacoby. John explained the changes’ purpose and provided the code. Everyone agreed they were beneficial changes, and he’s submitting code to WordPress for over a decade. Despite that, there would be 7 revisions, 32 comments, and 11 months before it was accepted. Many newbs get cranky if their patch takes a week to be accepted.
Lastly, if it’s your first time contributing code to a project, it’s likely you overlooked some aspect of the project’s standards for code or contributing. WordPress core has coding standards, but most projects have different ones.
Overall, getting a code contribution accepted without any discussion is very unusual. Expect there to be discussion and multiple revisions before your code is accepted.
Where The Real Learning Happens
This may sound a little discouraging, you would have probably preferred to hear “contributing code to someone else’s project is easy, and there’s never any need to discuss or revise anything.” But there’s a big up-side to all this: it’s where the mentoring by “world-class professionals” happens.
“Did you think about using x?” “What about users with y?” “If you do it like that, it’s a security vulnerability of type z.” I’ve found that it’s during discussing open source software that I learn a ton that helps me to be a better developer.
I could write an entire blog post on how to communicate effectively about open source software, but I’d like to just mention two tips.
Seek first to understand, then to be understood
-Stephe R. Covey
This is especially important in online, technical discussions. Re-read what others post at least twice, research everything term they mention, and try your best to understand the facts and why their priorities. What’s more, it’s often in the process of understanding the other person’s argument that you’ll either see its merits, or help them see its weaknesses. In honestly trying to understand others’ arguments you’ll build a better relationship, and lead to a quicker resolution.
When it comes time to be understood, try to make your comments not-to-long but not-to-short (don’t give a rant but do explain your argument well). Your argument is incomplete if you don’t provide an example, or a video, screenshot, or code sample. Provide links to pages that give further information.
If you can’t explain it simply, you don’t understand it well enough.
-“Albert Einstein” (supposedly)
What if you already understand the other person’s argument “perfectly”, and the problem is just that they don’t understand your argument? Chances are you don’t understand it well enough, otherwise you’d be able to explain it well enough that they would understand. In takes a great mastery of a subject in order to explain it simply. So even if you think you already know everything, just trying to explain it better and more simply will actually help you to master the subject even more.
(And generally, even if you have the most experience or authority on the subject being discussed, I still think it’s best to explain your reasoning or experience, rather than just say “this is better because I say so.” You may be right, but no one else will learn from your expertise unless you share it… only communicating decisions without any of the rationale is obviously faster in the short-term, but the long-term consequence is the learning is stiffled.)
An Improved Pull Request Discussion
If I were to have done my patch all over again, it would have looked more like this:
Rule 3: Be Positive and Grateful
The Value of Keeping It Positive
It doesn’t hurt to get on project maintainer’s good side. I now help develop and maintain Event Espresso, another WordPress plugin, and we get sometimes issues or pull requests from people who are just seething venom. We still try to act objectively and consider the merits of what they’re saying (not the way they said it), but it’s hard to think straight when you feel offended.
To have happy and healthy open source communities, we need to learn how to be smart about our emotions…
[Y]our community is made [of] humans not laptops. So assume good faith or better, and always communicate in a friendly way regardless of what your current emotions are…
When you’re having a discussion about someone else’s project, whether it be part of your pull request or just an issue, realize that you’re a digital guest. You are standing on someone else’s “property”, and should behave yourself, or else you’re liable to be kicked off. If you’re behaving like a troll (someone who just wanders around the internet stirring up trouble), you will be treated as such and be kicked out.
Chris Lema, a frequent WordCamp keynote speaker, has a similar policy regarding blog comments, which I think translates equally well to comments and discussions on open source projects
[You] can’t go to the outside of my house and spray-paint on my walls.
When you come into my house on the web, you can read all my posts, and you can write any comments. ..but you don’t get to decide whether I keep your comment and I make it public. I’m ruthless about taking down your trolling comments and getting rid of them on my site.
So when you’re a guest of someone else’s open-source project, realize you are entitled to nothing. You don’t get to demand things, or make them feel bad. If you do, you’ll quickly get the cold shoulder, or get shut out outright.
WordPress “Big Names” Always Keep It Positive
What’s more, having a bad attitude and acting entitled not only labels you as a jerk (and people avoid jerks) but also as a newb to the WordPress community.
If you watch the tweets between the WordPress big names, they’re always encouraging, thanking, and sometimes downright praising each other.
These tweets are very typical for WordPress big names:
It’s only the nobodies who are criticizing. (In truth, the WordPress big names sometimes criticize, but that’s more the exception than the rule.)
My patch for BuddyPress Docs wasn’t exactly negative -I really didn’t say anything- but it wasn’t positive, either. I never thanked Boone for the software he provided and maintained for free, nor did I thank him for his initial look at the code, nor did I thank him for his follow-up (although I did, only recently, do that on Slack).
Show Your Gratitude for Free Software
What’s more, if I wasn’t putting my “money where my mouth was”, I should at very least have put my “stars where my mouth was”. I mean, I should have given it a positive review on the WordPress.org plugin repository. We depend on free software for our livelihoods, and yet most don’t give a penny towards it, nor even a positive review online.
In the process of writing this, I finally rectified that mistake and gave a 5-star review for BuddyPress Docs. I now generally make that a habit for all plugins or themes I use (and apps, programs, and even restaurants I like). I’m invested in them and want them to succeed, it’s in my best interest to give them a positive review.
So far I think it’s helped too. When I try to stay positive and grateful for any help I get from project maintainers, it’s not only a good experience for me, where I often get what I was wanting, but I think I make a much better impression. Seriously, it seems only newbs are jerks.
Rule 4: Small Changes Are Easier
Less Code Means Less Discussion
The two previous rules pointed out there’s a lot to consider before and after a patch is submitted. This rule is a way to avoid a bit of that. If the change you propose is small, it will be easier to accept. While maintaining our plugin Event Espresso, we accept changes that fix a typo, add a missing closing HTML tag, or add a WordPress filter much more easily than we accept changes to how taxes are calculated.
Not all maintainers are keen on accepting massive change sets from new contributors.
The feature I was adding to BuddyPress docs was quite significant and so was unlikely to be exactly in-line with Boone’s vision for the project, and required a good deal of analysis.
So what can you do to keep your patches small?
- If your patch changes more than a few dozen lines of code, STOP! Check with the project’s maintainer if what you’re adding is what they want
- Make separate patches for distinct features/bugfixes (don’t put all your eggs in one basket, or your changes in one pull request, so that each one can progress unincumbered by the others)
- focus the patch’s purpose and avoid unnecessary improvements (e.g., if you’re fixing a bug, don’t try to squeeze in a new feature with it, or go fixing variable names in other parts of the code- do that on a separate patch)
- Don’t Repeat Yourself (if you are copying and pasting code more than once in different parts of the project, you’re repeating yourself; make a function or class that does that and reuse it)
How to Achieve Big Features with Small Patches
But how could I have made my change smaller while keeping all its features? I could have created a patch that only added a few WordPress actions and filters, and then put all my code in a new plugin that hooks into those actions or filters.
For example, in my patch, I added a bunch of HTML to show the form for handling the attachment uploads. The exact content of the code doesn’t so much matter, as just noticing its size. Here a portion of my patch:
There was quite a bit of HTML being added. Instead of putting all that in my patch, I could have simply added a call to do an action, like so:
If you were the project’s maintainer, which would you more easily accept? My original big change, or the tiny one-liner that actually doesn’t change anything (unless it’s hooked into). So what about my HTML? That needs to go somewhere. So we’d put it in all in a separate plugin, like so:
Again please note, that this last code snippet would only appear in my own plugin; so Boone would never even need to see it or consider it.
Had I created my patch like this, Boone would have needed to only review the added actions (which, if not hooked into, do nothing), which would have made accepting my changes dead easy. Plus, the same feature still gets implemented.
Rule 5: Companion Software is the Easiest
Make Your Own Project Instead of Adding to Someone Else’s
It’s possible to avoid all discussion with a project maintainer entirely, if, instead of creating a patch to add a feature to someone else’s project, you create your own project that extends their project. In WordPress, that usually means: instead of creating a patch to add a feature to WordPress core or one of its existing plugins, create your own plugin that adds the feature.
The Benefits of Companion Software over Creating a Patch
E.g., instead of trying to add file uploads to BuddyPress Docs, I could have created my own plugin that did it. This way, Boone would have had a lot less, if anything to worry about.
- Because no changes were made to BuddyPress Docs, it couldn’t introduce any security vulnerabilities or bugs, or slow it down, or add to Boone’s maintenance burden
- If I put the plugin on WordPress’ plugins repository, it would only be used by folks who wanted it; everyone else wouldn’t be bothered with it
- If my plugin became sufficiently popular, then Boone could consider merging that functionality into BuddyPress Docs. At this point, the feature’s value would have been demonstrated
And guess what happens once you’ve created companion software and made it available online? You’re now an open source project maintainer! Welcome to the club!
WordPress’ Companion Software Policies
That’s actually WordPress’ policy anyway: with few exceptions, all new features for WordPress core actually begin as “feature plugins”. Then, if they are sufficiently popular and useful, they will be merged in. That was how the WP REST API was developed and eventually merged into core; and how the new WordPress editor, Gutenberg, is being developed currently.
Another policy of WordPress is the “80% principle“:
The rule of thumb is that the core should provide features that 80% or more of end users will actually appreciate and use. If the next version of WordPress comes with a feature that the majority of users immediately want to turn off, or think they’ll never use, then we’ve blown it.
That is a common principle for most open source projects. Would 80% of BuddyPress Docs users want my proposed feature? I’m not sure, so it certainly would have been better as a separate plugin initially.
Separate Code is Faster Code
One last advantage of making a separate plugin is that it can be developed faster, and independently, from the project it extends. Why is that? Because if my plugin is used by myself, and maybe a few dozen other users, then making a change affects a much smaller group, and has less related software it needs to operate with (e.g., each server it’s on may be using a different version fo PHP, MySQL, web server, WordPress version, plugins, etc). Whereas if my changes become part of the main plugin, used by many more sites, each new change needs much more consideration and testing.
Recently someone created an add-on for Event Espresso, instead of submitting the code directly to us. The user aparna249 created Files Add-on for Event Espresso 4, which allows registrants to upload files while registering for events. They didn’t need to discuss the changes with us, we don’t have to maintain it, and they don’t need to worry about losing their customizations when Event Espresso is updated on their server.
And once you’ve created a plugin or other open source project, and make the code public, guess what you will be? A project maintainer. One of us.
Contributing to open source software will stretch you at first. But that stretching will not only mean better software, but you will become a better developer.
Just remember the five rules I mentioned:
- Justify your Changes
- Be Willing to Discuss and Revise
- Be Gracious and Positive
- Small Changes are Easier
- Companion Software is the Easiest
How to Fulfill Your Destiny
So what should you do now? Here’s a checklist:
- use open source software (like WordPress) and give it positive rating (assuming you want it to stay around)
- when you notice a deficiency, suggest an improvement (for most WordPress plugins and themes that’s on the WordPress.org plugin and theme repositories, but also often on Github, and for WordPress core its on trac.wordpress.org)
- become familiar with using the version control system Git and making a pull request on GitHub
- open a pull request to fix a bug or add a small feature for a project you use
- create companion software (e.g., make a WordPress plugin) to add more significant features to software you already use, and publish it on GitHub
- participate in a project’s discussion (for WordPress, this is on trac.wordpress.org and in the weekly dev chats held Wednesdays)
If you have experience working with open source, you have probably discovered other hidden rules that I missed. Please tell me in the comments!