This site best when viewed with a modern standards-compliant browser. We recommend Firefox Get Firefox!.

Linux-HA project logo
Providing Open Source High-Availability Software for Linux and other OSes since 1999.

USA Flag UK Flag

Japanese Flag

Homepage

About Us

Contact Us

Legal Info

How To Contribute

Security Issues

This web page is no longer maintained. Information presented here exists only to avoid breaking historical links.
The Project stays maintained, and lives on: see the Linux-HA Reference Documentation.
To get rid of this notice, you may want to browse the old wiki instead.

1 February 2010 Hearbeat 3.0.2 released see the Release Notes

18 January 2009 Pacemaker 1.0.7 released see the Release Notes

16 November 2009 LINBIT new Heartbeat Steward see the Announcement

Last site update:
2018-09-24 02:50:36

Proposed Release Testing Procedures

I propose that we have a test team of several people who are responsible for the testing of releases. This is obviously in contrast with the past procedures, where one person did most of the official release testing.

Key aspects of this procedure

  • Good communication
  • Testing will be done using CTS, and a test plan of manual tests, and ad hoc testing by the community
  • Not all testing done by test team, but the development community and Linux-HA community is expected to participate
  • Anyone with a cluster will be asked to participate, and spend much of their CTS resources on release testing during the release cycle.
  • Ask for volunteers to run the manual tests
  • Major stakeholders should feel free ask for releases near specific dates to make their releases easier. The test team should try and accommodate those requests when they can.
  • Anyone can nominate a bug as being a blocker, but if the test team disagrees, the person nominating and the test team are expected communicate with each other and the rest of the community to reach a consensus on how to handle it.
  • Test team will ultimately be responsible for creating and maintaining a schedule of releases – at least two and preferably three releases ahead of time. It is understood that this schedule may change, due to unforeseen circumstances, and keeping in mind that quality is master over schedule, not schedule over quality. In addition to these scheduled releases (3-4 times per year), there will undoubtedly be unscheduled releases to fix things that slipped by us.
  • Developers who will not be available during a test interval will announce it to the -dev mailing list as soon as they know it.

Communications

Although past release procedures have had their problems, the most persistent complaint about them (and a lot of other procedures) has been poor communication. Although this proposal has many places where communication is dictated, it is difficult to communicate too much during the planning and creation of a release.

Proposed Release Testing Procedure

During the testing of a release, will follow a plan – which they will announce in advance. The test team is expected to announce any deviations from the plan to the community as they go along.

I propose dividing the testing interval into two phases – initial and final testing intervals.

Initial testing interval

  • Pull 'dev' into test
  • Begin initial testing – CTS and manual according to test plan, and ad hoc testing by the community.
  • Report initial bugs
  • Weekday daily test cycle
    • Incorporate bug fixes into 'test' for problems found during testing
    • Bug fixes are pushed into 'test' with no more than a day's delay during the week, and until Monday on the weekend
    • Developers who want bug fixes to be included in the next release will email the test team to pick them up for the next release. It is expected that fixes be given to the test team in the 'dev' tree. Requests to have bug fixes included will be given explicitly by email, as implicit schemes based on descriptions, etc. are unreliable. The test team will in turn reply to such emails acknowledging that they will pick up the fix.
    • Each tester should run about one day's worth of testing each day and then pick up the next day's version, incorporate it and test it.
    • Package builds for 'test' are made available for major platforms (using Andrew's build process on OpenSUSE?)
  • Repeat for a few weeks (2-4?). More for releases with major features, and fewer for releases which are contain only bug fix and minor features.

When the test team thinks it's close enough, then it will announce “final testing interval” to both the -dev and the main mailing lists. Although there will be a schedule for entering the final test interval, quality is master over schedule, not schedule over quality.

Final Testing Interval

  • During final testing interval, bug fixes are restricted to only include regressions, and fixes that affect data integrity. The decision on including a fix will be made by a consensus between the test team, the developer and the community. In the unlikely even that they cannot come to a consensus, we'll have to figure out how to handle it if it comes up.
  • Each new update pushed to 'test' during final testing interval will be announced to the -dev mailing list. Of course, the usual RSS feeds for Hg will be available. However, since communication has been one of our failures in the past, we will communicate by both RSS and email – so that no one is missed.
  • Any non-trivial fix will cause the final testing interval to restart. During this final stage, developers must state if a bug fix is trivial or not based on their knowledge. If the test team disagrees with respect to their place in the testing procedure and their knowledge of the fix, they will dialog with the developers involved to reach a consensus.

Activities:

  • Developers and test team and volunteers test 'test' base using CTS
  • Test team and volunteers execute test plan against 'test' version
  • Test team and volunteers report bugs to community Bugzilla
  • Final testing interval shall include complete manual tests from test plan, and at least 5000 iterations of the CTS suite spread across at least two architectures, preferably more.

Responsibilities

This section outlines the responsibilities of each of the set of participating stakeholders in the Linux-HA project during the official test intervals.

Developer Responsibilities

  • Communicate status, progress, and problems with other stakeholders using the linux-ha-dev email mailing list, and when appropriate, private emails, conference calls, and other phone calls.
  • provide timely bug fixes for bugs assigned to you to the 'dev' workspace
  • avoid committing major restructuring and similar high-impact things to 'dev' during the entire testing interval.
  • Build RPMs using make rpm and run BasicSanityCheck before pushing any fixes to 'dev' during test interval

  • Run CTS tests during test intervals as specified
  • Run manual tests – from test plan and/or ad hoc
  • File bug reports for discovered bugs – making sure bugs get assigned appropriately
  • When creating bug fixes during testing period, email the -dev list asking for your fix(es) to be incorporated.
  • Verify that bugs you submitted are fixed after they've been incorporated into 'test' workspace.

Test team responsibilities

  • Communicate status, progress, and problems with other stakeholders using the linux-ha-dev email mailing list, and when appropriate, private emails. Status communication regarding high-priority bugs, missing fixes, fixed bugs, etc. is especially important during the test interval.
  • Publish overall schedule for the stages of the testing progress based on Alan's draft. Review, correct and update this schedule. After the first release, the test team will create future schedules.
  • Develop and publish a manual test plan on the wiki site for testing, participating in discussions with, and incorporating feedback from the community. This test plan must include limited testing for R1-style clusters, and also for upgrades as well as initial (clean) installs.
  • Manage the 'test' workspace
  • Incorporate bug fixes to the 'test' workspace according to the rules for the different phases of the test interval
  • Carry out test plan – both manual and automated testing. Make sure that someone is doing at least limited r1-style testing. If you don't do it, find someone who will do it for you. 500 iterations during end of initial test cycle, and 500 during final test cycle is sufficient.
  • Check with other community members to ensure that we are covering these platforms at least minimally during testing:
    • x86 linux
    • x86_64 linux
    • ppc linux
    • s390x linux
    • x86 FreeBSD
    • x86 OpenBSD
    • x86 solaris
  • Create bug reports for bugs discovered while testing. Make sure bugs are assigned to an appropriate person.
  • When release is completed, tag released version
  • After tagging release, push to the stable repository
  • Tell Alan to publish release and digitally sign tar ball, packages, etc.

Community responsibilities

Communicate status, progress, and problems with other stakeholders using the linux-ha-dev email mailing list, and when appropriate, private emails, conference calls, and other phone calls.

  • Evaluate and comment on test plan
  • Perform ad-hoc testing during entire test cycle
  • Perform BasicSanityCheck testing during test cycle

  • Offer opinions as appropriate on the severity of bugs and the relative importance of bug fixes in your environment and experience
  • Perform CTS testing (if possible) during the testing cycle
  • Make sure your OS and/or distribution works well with the new release
  • File good, detailed bug reports as appropriate. Dejan Muhamadagic has a tool which helps with this.

Miscellaneous responsibilities

These responsibilities belong to individuals and small groups of people.

Alan Robertson

  • Communicate status, progress, and problems with other stakeholders using the linux-ha-dev email mailing list, and when appropriate, private emails, conference calls, and other phone calls.
  • Participate as community member and as developer – as appropriate.
  • Coordinate testing on IBM hardware
  • Announce this document to the community
  • Draft initial proposed schedule for the first release following this document. This will be handed off to the test team for their review, update, and correction. The test team will publish future schedules.
  • Participate in discussion concerning this document, and evolve it according to the community consensus.
  • Publish final release up on the project web site
  • Announce new release to mailing lists, to freshmeat, LWN, etc.

SuSE/Novell folks

In addition to your responsibilities as developers and/or community members, someone needs to make sure these things get done.

  • Communicate status, progress, and problems with other stakeholders using the linux-ha-dev email mailing list, and when appropriate, private emails, conference calls, and other phone calls.
  • During initial testing interval: produce nightly builds from the 'test' image
  • During final testing interval: produce builds corresponding to what the test team is testing (might be nightly builds)

Open Issues

  • How will release notes and change log be created, and who will do it?

Testing outside Release Intervals

We need to automate the following things against the 'test' tree daily to make the final test interval go smoothly.

  • Nightly builds and nightly BasicSanityCheck and CTS runs with automatic emails to the mailing list on failure. This should include at least i586, x86_64, ppc, and s390x architectures.

  • Ideally we'd like to do something nightly for *BSD, Solaris, and OS/X – assuming we can get volunteers in the community to set up cron jobs for us
  • Scans by automated verification tools (Coverity (Stanford Checker) and/or BEAM)

If these things are done daily, then it is much less probable that new problems will be found during official testing. The sooner a problem is found the cheaper it is to fix. This will greatly smooth out the release process.

SEE ALSO

Release 2.1.3 schedule