I love reddit.com, who doesn’t? I found this awesome IAmA (I am a…) today and thought I’d share it in an interview form here. Enjoy!
Question: Out of curiousity, what CMS are you using on mozilla.org, the Firefox site, support.mozilla.org, et al.? You guys have an awful lot of pages to manage.
Most of our websites nowadays are based on Django, and the content is not usually user-editable. We have a Django-based framework called playdoh that we use for almost all our websites, with a few notable exceptions.
One of those exceptions is We actually don’t use a pre-made CMS on mozilla.org. Instead, it is currently a slim PHP framework with lots of pages. Here’s a technical description of how this is glued together. However, since that’s a legacy thing and somewhat hard to maintain, we are in the process of switching all that over to a Django-based site as well.
Question: What’s your deployment strategy ? What do you test in your codebase and how ?
We also have developed a culture of testing–this is something I’ve been meaning to write a blog post about. That means a few things:
- Time to write tests is included in how long it takes to write the code under test. A feature isn’t complete without tests.
- If you break the tests, there’s some good-natured teasing, and you lose points in the CI game. Light social pressure is incredibly helpful.
- If you break the tests, your first priority is fixing them.
Next I want to develop a culture of performance.
Deployment - This is my favorite topic! I’ve been giving talks on it for around a year now. I actually started putting a joke about that into my talks about it.
My goal with all the projects I touch is to deploy continuously. Not only does continuous deployment mean fixes get to users as fast as possible, but it has a bunch of requirements that are great in-and-of themselves, like…
- You must have a robust, automated, and fast deployment pipeline. One-button and wait.
- You must have a high confidence level from automated tests so you don’t break things.
- You must have active, real-time monitoring of the site.
- You must keep master/trunk/whatever branch in a clean, working state, all the time.
- Developers must develop a sense of ownership over their code that lasts all the way out the door.
It also has a number of side benefits, like not doing code pushes at night when people are tired, or about to leave for the day.
Some projects are closer than others. We learned a lot from Etsy (link to blog post and video, watch Kellan and Erik’s sections). But their Deployinator tool is Ruby, and it took them a while to open source it, so we built Chiefto do the same thing (run some shell scripts, print a bunch of output).
We’ve got push-button production deploys with Chief or other, ad-hoc tools, for a few sites now. We’ve got it set up in the -stage environment Fred mentioned for a few more. We use another tool called Freddo to deploy -dev environments on Github post-push hooks. My goal is to have all new environments set up with Freddo and Chief by default in the future.
Our deployment strategies vary a little bit between projects, but as a general idea, we have three (sets of) servers: -dev, -stage, and production. When you commit something to the codebase, it automatically updates -dev. -dev always represents trunk and it thus a moving target, and not too stable for manual testing. Once a release is code complete, it is pushed to -stage, which will then remain like this until the release. QA tests on stage, and once we are confident that we can release, we update the production servers. For comparability, the stage setup is supposed to be a smaller version of the production servers, but with the same hardware and software setup (so, for example, if production has 4 databases and 10 webservers, then stage would have 2 databases and 2 webservers).
Testing is a big deal at Mozilla: We (try to) unit test everything, and have a continuous integration server that gets angry at us when tests break. Our test framework is the one that’s built into Django, along with nose and a few of our own extensions. When doing code reviews, we reject patches that are untested. Once that tested code has landed, we also have a very capable QA team (and contributors!) that focuses on integration testing as well as test automation for “user flows”. For example, “can a user upload an add-on on AMO?”. QA has their own CI server that also complains when something breaks. The tool we use for this is Selenium.
Question: As I heard you use a lot mysql. Since mysql is oracle stuff now, you have any intention to moving to another DB like postgres? And what about NoSQL the webdev team had the intention of using a db like mongodb?
We’re using MySQL for a lot (not: all) of our webapps, that’s true. As long as MySQL remains an active open source project, there’s no reason for us to abandon it based on who owns the company maintaining it. We also have two DB admins on staff who are really experienced with MySQL (but not only).
That said, we are using Postgres for some projects already and will continue using it whenever it’s good for a project. Same goes for “NoSQL” databases: We’re using Redis for on AMO, for example, and a number of us have used MongoDB on smaller projects before, though not on big Mozilla projects.
In general, we strongly believe in using the “right tool for the job” and are thus open to new technologies whenever they make sense.
Question: Don’t you feel that the Chrome dev team is leading the web forward (with the exception of some evil moves by Google like Dart) like you were before ? If so what, in your opinion, are you missing ? Developers ? Money ? Also, is Lion compatibility a priority, and how many contributors/developers are on it ?
I as a web developer do think that Chrome is moving the web forward in certain ways. And that’s a good thing! However, make no mistake that Mozilla and Firefox is working any less hard on advancing the web. Just one example, here is a meta bug on all the “new web APIs” we are working on.
Good question, sadly I don’t know how the team size compares. However, I am fairly certain that Google’s teams, in every aspect of what we do, by far outnumber Mozilla’s team sizes. So resources are definitely an issue.
As for the motives, I agree that the individual developers on Chrome probably share our views of advancing the Open Web.
Do I feel like Chrome is a leader in innovation? Most certainly. Do I feel like Firefox is another leader? Absolutely. Chrome’s number 1 distinction point is speed, while Firefox’s is user experience and privacy, for example. Luckily, there is not only one way to advance the Web.
Question: What are your views on Node.js? Would Mozilla considering switching to Node.js? ( Bringing ALL of the products, Firefox, Thunderbird, and website frontend to be made with js ) Would you stop using node.js because the Engine is made by a competitor ( V8 from Chrome )
Node.js is great! There are a few things that we’re already doing in Node. The most well-known one is probably the backend to BrowserID (not a webdev project, mind you). That its JS engine is V8 is not an issue.
That said, switching over all projects to node does not seem to make a lot of sense. I don’t think Node is (currently) a good all-purpose platform, and we don’t need its specific strengths for all our projects.
For example, I have an iGoogle-like project on the drawing board, but it’s likely that I’ll have to use either Django+Node.js or Tornado or Bottle+gevent because the few RSS/Atom-parsing libraries I’ve found for Node.js are immature (Everyone’s focused on JSON-based Activity Streams) while Django and the WSGI API weren’t designed with server-push and long-lived connections in mind.
Question: How many of you work remote – and do you think that it makes the job easier/harder?
Mozilla is a very, very remote-friendly company in general, and our team is (I think) remote as a majority. Personally, I’m hooked on it and would have a hard time going back to an office, now.
IMO, the “easier” part is being able to work without outside interruptions, but the “harder” part is being self-motivated enough to work without interrupting yourself. My home office is mere steps away from a fully-stocked entertainment center with an Xbox. Of course, the office in Mountain View has that, too, but at least there my co-workers can see me slacking off
There’s also the issue of communications – but we attack that problem with documentation, IRC, IM, VoIP, video chat, and whatever else we can come up with.
That’s actually a good thing to attack, too, because we make Open Source software. Barely any contributors from the community have the advantage of being present in a Mozilla office, so being remote helps to keep Mozilla employees on the same level with non-employee contributors because most of the communication tech we use is not exclusive to the company.
(For instance, did you know that our weekly all-company meeting is a public conference call? Every week.)
I think more than half of us are remote, probably around 2/3rds. Most of us remoties work from home. (Some people are in a Mozilla office, but are the only webdev there, which is kind of like being remote.)
I think it has challenges. We get together in meatspace pretty regularly, because it’s valuable and some things are easier solved with no communication lag time. We also do a lot of phone and video conferences for that reason.
But once you have a critical mass of remote people, it starts to get easier. There isn’t a stigma to being on the other side of the webcam or phone, if 2/3rds of the people are. Even folks in conference rooms at the office are usually talking to a camera. And we have to put things in asynchronous tools like email, wiki pages, github comments, which also makes it easier for us to include community members.
EDIT: On a personal note, working from home is about a wash with being in the office, having done both. I go a little stir crazy, but I have a more flexible schedule. I don’t see people during the day, but that makes me have a more active social life at night and on weekends, etc.
Question: What should a upcoming Mozilla webdev intern know before coming in?
This question implies that you have already been accepted as an intern (correct me if I’m wrong), so that would assume you have shown that you can work in a team, are interested in the things we are doing, and that have written some relevant code.
You are also welcome to pick any Mozilla website you find interesting and take a look at its code, ask questions, maybe even file a bug or submit a patch. Also, go ahead and find us on IRC (http://irc.mozilla.org for instructions, channel #webdev) and say hi!
Looking forward to meeting you soon