• Testing the Open Web Platform 

    As the Web has grown from a documentation sharing platform targeted essentially at desktop users to a fully applicative one available on all kinds of devices, so have its testing requirements. The test suites published by a number of W3C Working Groups in order to move their specifications along the Recommendation track are no longer sufficient. The kinds of applications built by developers require a level of conformance, quality of implementation, performance and interoperability that can only be guaranteed by much more thorough testing. In truth, the testing requirements are such that a concerted effort involving the industry as a whole is necessary in order to meet them. Unfortunately, so far, our infrastructure has been cruelly lacking, which has really impeded any such effort.

    But this is about to change.

    W3C is launching an unprecedented effort to scale up its test offering. And the good news is this effort is backed up by significant financial and human contributions from the W3C Membership. For the next year, my employer, Facebook, will be covering my salary and expenses while I focus, full time, on driving this effort at W3C. Some W3C Members have already committed human resources, others are poised to contribute tests or significant financial support.

    Over the course of the next year, we’re going to take a number of steps to improve our testing infrastructure and documentation, simplify our processes, and increase test coverage.

    First, we’re going to move all tests to a central repository on GitHub. This will be an opt-in process for Working Groups, so don’t expect change overnight. A number of Working Groups have already moved or are about to do. Other groups may require additional time, especially if they have invested in infrastructure built on top of Mercurial. There will be migration costs, but I’m positive that the benefits of a common repository, tooling, and processes will outweigh them.

    Second, we’ll be streamlining test contribution and review processes. Writing a test, getting it reviewed and approved shouldn’t be more complicated than forking the repository on GitHub and making a pull request. There will still be a contribution license, but the process for agreeing to the license will be simpler than it is now.

    Third, we’re creating a dedicated resource center for all things testing. There, you will find documentation, tutorials, test coverage, browser support, and other valuable data points, as well as information on how to download and run the test suites locally.

    Fourth, we are overhauling the test framework to cater for the different use cases we’ve collected, such as running subsets of test suites or being able to install a local copy of the framework to test prototype devices or unreleased browsers. We will also set up an instance of the test framework on a public server so you can run tests on various devices over the Web. And we’ll publish test results to give developers a clear picture of support for a feature (e.g. IndexedDB) or even a subset of a feature (e.g. blob support in IndexedDB).

    Where will the tests come from? We are investigating more direct solutions to increase test contributions and speed up reviews. Browser vendors are key contributors in this area and we’ve started discussing ways to simplify upstreaming their tests and syncing to our test repository. That's not a trivial undertaking, so it could take some time before any of this is in place. Once set up, however, this could be a significant step forward.

    Developers have also become an important source of test cases, thanks to great efforts such as Test the Web Forward. We’d like to see this trend continue and to foster a strong developer community around testing. I hope that the full slate of improvements we are making will move us in that direction.

    It is also conceivable that we outsource the development of tests for some parts of the platform, for example to meet a particular timetable (such as HTML5 becoming a Recommendation in 2014). The changes described above will also help us manage this scenario more effectively.

    There’s plenty more we’d like to tackle in the future (integration testing and performance benchmarking immediately come to mind), but I think the above focus is a solid starting point. I invite people to follow our work on the wiki and our mailing list. If you or your organization want to help us in any way, please contact me either directly or through the mailing list.

    This journey 1% finished.

  • Test The Web Forward Sydney! 

    Mark your calendars!

    Test the Web Forward is bringing W3C testing goodness to Sydney, Australia on February 8th and 9th, 2013.

    The Sydney Test the Web Forward event is the latest event in a movement that focuses on learning, hacking, and writing tests for W3C specifications.  During the event, experts will teach you about W3C specs & W3C testing and will guide you as you help make the Web a better, more interoperable place. Better tests for a better Web!

    To learn more about the past events, check out our posts on Test the Web Forward ParisBeijing and San Francisco.

    Get Registered

    Registration is now open for the event on Eventbrite.  

    More details for the Sydney event can be found on the Test the Web Forward website and follow us on @testthewebfwd for updates. See you there!

  • Chinese Developers endeavor to get more involved in the community - Test The Web Forward hackathon in China 

    W3C China Office and Adobe co-organized the Test The Web Forward event on Oct 20-21, 2012 in Beijing. The intention of this hackathon was to help more Chinese developers to get involved in contributing to the web platform. Three W3C specifications including CSS Flexible Box, CSSOM View, and CSS3 Backgrounds and Borders were targeted during this event. 17 experts from W3C, Adobe, Google, Intel, Mozilla, Opera, UCWeb, Tencent, Baidu, Taobao, Qihoo360, Maxthon, and Huawei were there to help the attendee to get acquainted with how to Read a W3C Spec, how to Write a W3C Reftest and how to File a Good Bug Report. Over 260 developers from about 160 organizations registered for this event. 119 test cases (hard: 1, normal: 9, easy: 109) were submitted and 12 browser bugs were filed as the results of the test hackathon. Positive feedbacks have been gained from local developer communities and the industry. And the enthusiasm to continue the efforts of Test The Web Forward event next year in China is under discussion and more local industry volunteers to participate and contribute.

    Details about the browser bugs that have been filed during the event:

    CSSOM View
    • Firefox Bug 803950 - matchMedia addListener event cannot be triggered every time by width be change over the condition.
    • Not yet logged into bug system] Opera alpha channel bug
    CSS3 Backgrounds and Borders
    • WebKit Bug 99922 - [css border] border-image doesn't honor broder-style
    • WebKit Bug 99926 - Should show complete tiles when set 'border-image-repeat' as round
    • WebKit Bug 99927 - transform3d conflict with background animation
    • WebKit Bug 99928 - [CSS] shadow from radius has wrong render in webkit
    • Firefox Bug 803947 - [css border-radius] replaced element set border-radius will contents overflow.
    • [Not yet logged into bug system] IE Bug: When IE9 table's border is set to collapse, border-radius is not ignored.
    CSS Flexible Box
    • WebKit Bug 99923 - margin-top/bottom has no effect for child nodes of flex items
    • WebKit Bug 99924 - WebKit does not support 'flex-wrap: nowrap'
    • WebKit Bug 99925 - Setting width of a flexitem causes the adjacent flex item to be displayed poorly.
    • [Not yet logged into bug system] Bug: Flexible Box Override child display property and set them to display:block
  • Test the Web Forward Paris! 

    On October 26-27, the third successful Test The Web Forward event was held in Paris, France. Over 100 local web developers joined 35 W3C experts for a fantastic weekend of  learning, hacking, and writing tests. The momentum is now undeniable and the enthusiasm from the attendees and experts at this event was palpable.

    Friday Evening - The Talks


    Vincent Hardy begin the evening delivering l'accueil chaleureux to a room full of energetic and passionate web professionals. With Test The Web Forward Beijing recently behind us, he gave the group an update on the success of that event.
    Vincent Hardy
    Vincent laid out the agenda, which included a series of short talks designed to provide newcomers with the info needed to get started writing tests the next morning.

    How to Read a W3C Spec


    Robin Berjon from the W3C kicked off with How to Read a W3C Spec.
    Robin Berjon
    It was a great presentation on what might have otherwise been a very dry topic. With Robin's flair, it was not only very informative but highly entertaining and included great visual aids.  To illustrate the point of why we have specs, he used two slides of a "red button" showing the button spec on the left and the implementation on the right.

    How to Write a W3C Reftest

    Onto the test writing instruction, I stepped up and presented on How to Write W3C Reftest, a similar talk that I also gave at the SF event. The presentation was step by step tutorial for writing an actual test, using the CSS Transforms spec as an example.
    Rebecca Hauck

    Testing With testharness.js

    James Graham from Opera & author of testharness.js followed with a great walkthrough of how write tests with it.  His talked piqued a lot of interest as was seen in the number of questions during and after his talk.  Also, we had a LOT of JS tests produced the next day!
    James Graham

    How to File a Good Bug Report

    Lastly, Elika "fantasai" Etemad gave her excellent talk on How to File a Good Bug Report.  She also gave a version of this talk at SF and she also got a lot of good comments from the crowd when she finished.

    The Lightning Talks

    We then did a series of lightning talks where a few experts got up and in 5 minutes made a case for tests to be written for specs or projects they are passionate about:
    • Michael Brooks had a very cool hacking project planned to convert the Apache Cordova/PhoneGap MobileSpec test suite to testharness.js tests, enabling them to be contributed back to the W3C
    • François Remy from the CSS WG called out that the CSS Values & Units spec is in CR and really needs tests – especially for the new viewport units
    • Arron Eicholz from Microsoft also mentioned that the Backgrounds & Borders spec is in CR and also needs some love
    • Kris Kreuger from Microsoft & HTML5 WG challenged people to jump in and write tests for the behemoth HTML5 spec
    • Odin Hørthe Omdal from Opera then got up and pitched IndexedDB and pointed people to all the materials he provided to get them started
    • Lastly, Patrik Höglund from Google excited folks about writing tests for WebRTC with a quick demo
    Michael BrooksFrançois RemyArron EicholzKris KruegerOdin Hørthe OmdalPatrik Höglund
    The lightning talks were a new part of the program and were a great success, evident by the fact that each of these areas had lots of great tests written or converted the next day.  In general, everyone finished Friday evening with a good idea of what they wanted to hack on, how to get started, and who the right experts were for each area.  A quick show of hands before we broke for beer & pizza showed everyone had a plan.

    Meet the Experts

    For the remainder of the evening, the attendees and experts mingled and strategized on which specs to target the next day.
    Meet the Experts

    Saturday - The Hacking


    Areas of Focus

    We divided the testing efforts into 3 categories: HTML (&WebApps), CSS, and Mobile.  In each of those, we highlighted several specs or projects:
    We were delighted that many more specs were tested in addition to those highlighted:  DOM Events, EventSource, URL, CSS Transitions, Filters, Regions,  and Multiple Columns.
    In total, this very diverse community wrote tests for 13 specs! Plus, an entire vendor suite was converted and now ready to be submitted to the W3C!

    Tests Written

    Hilarity ensued when the preliminary number of tests written tallied up to 404!  However, and like all good testers, we uncovered the bugs and adjusted the total to an even more impressive grand total of 499 Tests!!
    Here are the numbers broken down:
    HTML5 & WebApps – 139 total
    - HTML5 spec: 42
    - WebRTC: 29
    - IndexedDB: 62
    - EventSource: 1
    - URL: 2
    CSS – 214 total
    - Transitions: 95
    - Values+Units: 54
    - Transforms: 22
    - Backgrounds+Borders: 24
    - Filters: 2
    - Regions: 13
    - MultiCol: 4
    Mobile – 146 total Apache Cordova tests converted
    In addition to the impressive number of tests written, several people also hacked on some test infrastructure including improvements to testharness.js, the WebIDL harness, and adding support for test widgets in W3C specs.

    Favorite Quotes


     "Until we came here and did this, we didn't even know where we stood testing-wise"
    - Henrik & Patrik, WebRTC Experts
    "We have ~12K tests for HTML5. But we need like 1 million. So get crazy and scatter shoot tests at whatever you find in the spec“
    - Tab Atkins
    "I really want to do this more. Who do I talk to about getting more involved?"
    - Attendee



    Special Thanks

    As with all grass roots movements, this would not have been the success it was without the help & contributions from our esteemed community of experts. Thanks to this great group of people who prepared, traveled, promoted, and came to support the newcomers and hack some tests! --  Glenn Adams, L. David Baron, Robin BerjonMichael BrooksRik Cabanier, Julien Chaffraix, Arron EicholzEmil A.Eklund, Elika EtemadSylvain Galineau, Israel Noto Garcia, Daniel Glazman, James Graham, Patrik Höglund, John JansenHenrik Kjellander, Kris Krueger, Tobie Langel, Peter Linss, Divya Manian, Larry Masinter, Larry McListerAndre Jay MeissnerOdin Hørthe Omdal, Jérémie PattonnierSimon PietersFrançois RemyDavid Rousset, Dirk Schulze, Alan Stearns, Lea VerouJet Villegas, Levi Weintraub, Mike West, and Steve Zilles.
    Also, thanks to all the sponsors: Google, Facebook, Microsoft, Adobe, Telecom ParisTech, W3C, Zenika.
    Extra special thanks to Cyril Concolato at Telecom Paris and to the entire Zenika team for planning such a great weekend.
    Zenika Team
    ... and to Al Amine Ousman at Zenika for this beautifully produced video that captures the weekend perfectly:





  • Test the Web Forward Beijing! 

    Last weekend, on October 20th and 21st, the China web community got together to Test the Web Forward at the Jinma Hotel in Beijing.

    China Web Community

    The China web community is very diverse. According to An Qi (Angel) Li of the W3C China Office there are over 400,000 web developers in China, and the number is growing. At the Test the Web Forward event, there was a good representation of some of the major developers and companies that make up the community, including three local companies that make their own desktop and/or mobile browsers.

    There was an average of 100 people who attended over the two days of the event, from 96 different companies. The presenters and experts who participated in the event included two key web advocates, Winter and Xiaopang Dacheng, as well as people from 360, Adobe, Baidu, Google, Huawei, Intel, Maxthon, Mozilla, Opera, UC Browser, Tencent, and Taobao. It was amazing to have such a great turnout to Test the Web Forward, especially since it was the first event of its kind in China.

    Focus Areas

    Test the Web Forward in Beijing focused on expanding testing coverage for three specifications specifically: CSSOM View, CSS3 Backgrounds and Borders, and CSS Flexible Box. The event started on Saturday afternoon with an overview session for each specification, so that participants could get a better idea of what they were going to test. Johnny (Jianning) Ding from Google did an overview of how to read W3C specifications, and then reviewed the CSSOM spec. Mark Dong from Adobe reviewed Backgrounds and Borders, and Kang-Hao (Kenny) Lu from Opera reviewed Flex Box. Zhiqiang Zhang and Elva Fu from Intel China wrapped up the day with sessions on how to create tests for the W3C test suite.


    On Sunday, the test hacking began. In the San Francisco event earlier this year, a gong was used to help celebrate when a new test was submitted, reviewed, and approved. In Beijing, we used a Chinese tanggu drum instead to get people motivated to write tests.
    The beats started sounding pretty quickly as Angela Wang from Centilin Technologies had the first test of the event approved, for the background-clip-image property from Backgrounds and Borders.

    All in all, the event resulted in the creation of 119 tests! And with those tests, 12 new bugs were found and submitted across multiple browsers.


    Although learning new things about the web, and making contributions to make the web better was sufficient to get everyone excited, some awards were added to make it even more fun. Prizes included an iPad as well as gift certificates to local stores.

    Most tests written

    Zijun Wei won first place with 24 bugs written.

    Best bug

    Shaofei Chen found a bug in WebKit where margin-top and -bottom have no effect for child nodes of flex items. It’s already been fixed!

    Best test case winner

    Weisong Hu wrote the best test case, testing the flex item child margin.

    Next up, Paris!

    There were over 200 notes and comments on Sina Weibo (a Twitter-like microblogging service for China) talking about Test the Web Forward Beijing.

    It was great to see such enthusiasm by the people in the community and those who attended. It was clear from the discussions we had that there's a desire for more events where the community can get together and collaborate, and people were already starting to talk about the next one. The next Test the Web Forward event in China may happen as early as the beginning of next year - so stay tuned.

    In the meantime follow @testthewebfwd for the latest news, and contact us at testthewebforward@gmail.com with any feedback or suggestions. If you can’t wait until the next China event, head over to Test the Web Forward in Paris which happens this Friday and Saturday, to contribute to a better web!