What is experimental release?

I think our experimental build category may confuse some people. Our message about experimental releases is mixed – we like people to download and use them, but warn that they may cause data loss!

What exactly are these experimental releases? I covered reasoning for experimental releases in my earlier post. Experimental releases are simply one key factor behind WinMerge success.

It all started quite a bit differently. Several years ago we only did rare stable releases and some beta releases between them. Then we wanted to make some custom builds to test certain patches. People testing WinMerge usually don’t care about compiling it (and don’t even have the tools). So I started adding some custom builds to my own homepages for people to download and test. That worked for a while for some patches, but obviously it didn’t scale and very few people found out those custom builds.

So we added new release category for these custom builds. Usually we included patches that weren’t in CVS yet to test some new features. At that time Perry and Laoran were building and releasing these custom builds. I didn’t even have release rights back then. And I don’t even remember the original release category name.

Later somebody started doing snapshots builds from from current CVS code, without any additional changes. That is, he checked out latest sources from CVS, build executables and uploaded it as a release. I don’t remember who it was originally. Back then there was between 10 and 30 downloads for each release. But we were very happy about that! We got many bug reports and fixed them so our actual releases were getting better. Later I took the responsibility of making WinMerge releases. First I released only these experimental builds for people to test (as others were too busy). Then started doing also beta and stable releases.

This is how I make experimental releases today:

1. Check that there is not critical bugs open in bug tracker (so the release is generally safe to use).
2. Check out current codes from SVN
3. Build a executable for fast testing – do couple of test compares to see at least it starts and compares something. I really don’t do any real testing for experimental releases.
4. If I don’t see any immediate problem, I’ll cleanup possible local changes, set version numbers etc and do a new builds of executables. In practice my create_release -script does most of the task in creating a release.
5. If I’m lucky, there are no any compiling problems, missing files etc. So I can just zip the files, run virus check and upload them to SourceForge.net as a new experimental release.

That’s it! Criteria for experimental release is:

• current sources build (or are easy to fix by me)
• no immediate problems visible, meaning I don’t see anything weird by comparing couple of my test folders

Yes, the word testing is very much absent. There is no such thing for experimental releases!

Above is the very reason our User’s Guide says about experimental releases:

Use with care and enable backups!

And release notes say:

Use 2.8.0 stable version unless you are ready to tolerate crashes, possibly losing unsaved data etc. You have been warned.

These warnings are not without a reason. Most people just ignore them, and unfortunately many don’t even see these warnings as they download experimental releases from some download sites around the web.

I don’t remember there really being really bad data loss bugs in any experimental releases. Usually those kind of bugs get caught by developers already. But bad bugs happen in experimental releases. Couple of time I’ve had to remove a release pretty quickly after such a bug has been reported so that more people don’t get affected. The positive side of course is, the bug got reported, fixed and soon we’ll have a new fixed release. This is how we get better releases.

In summary:

• we do experimental releases to allow interested people to test WinMerge’s new features and other changes
• we do not recommend using experimental releases in production.
• do not expect experimental releases to be same quality than our beta- or stable releases
• be careful! Make backups of your data!
• report any bugs you see in the release! (Check first that it is not already reported.)

Experimental release!

2.8.0 stable was released almost three weeks ago. It was released almost an month after 2.8RC. So its been a while without experimental releases. But now they are back!

The 2.8.0 release made me finally realize how important these experimental releases really are! There’s been no significant bugs reported about 2.8.0 release. Which is just unbelievable. Over 70 thousands of downloads and no big bugs reported. With earlier 2.x.0 release there’s always been several bad bugs reported.

Yes, there are several bugs reported about in-line difference highlighting. But it was expected, we know there are bugs in that feature. There just isn’t anybody around to fix them. (Hint, hint!) But there are no bugs like WinMerge crashing comparing certain files, not finding all files, telling wrong dates etc. Which makes 2.8.0 look like a good release. Much better than I could hope for.

Releasing all those experimental releases hasn’t always been fun. It always takes some time and effort to make a release. And sometimes there are last-minute fixes needed as something does not compile or something was missed in earlier commits and so on. And then there is people’s expectation that experimental releases are good releases! And I keep repeating that they are just snapshots from our version control. They can be totally broken. But the fact is they’ve been good ones and people are get used to that – to not expect problems using them.

I think that expectation of certain quality has lead to 1000-2000 downloads per experimental release. Just imagine what kind of tester group that is! No, I don’t say all those users are downloading releases for testing it. They use it, perhaps for their day work. But at the same time they help testing. They use it (a lot, I hope) and report bugs when they find them. I don’t expect we hear about all bugs, but I’m pretty sure we hear about bugs that prevent doing a work with WinMerge.

The net result of all this is: we did some pretty major changes to the WinMerge between 2.6.0 and 2.8.0. Yet we don’t seem to have any big issues in 2.8.0 release. They all have been reported by people testing those experimental releases. So we could fix them before a major release. This for sure motivates me to continue releasing experimental releases.

There is one downside too: making big changes to WinMerge is harder. We should be able to release good experimentals while doing those big changes. It is just not possible. So there will be breakages. But I sure hope people understand (and have read all those warnings!) and keep downloading.

WinMerge 2.8 is out for real

Yes, WinMerge 2.8.0 stable release is really done. Get it from SourceForge.Net downloads: https://sourceforge.net/project/showfiles.php?group_id=13216&package_id=11248&release_id=589386

Our web site is not up to date and release announcements are not done. It’ll takes few days to get all things in place.

So what happened? Doesn’t release mean we need to have everything set up and ready? Yes, definitely. And we thought we’ll have them in few days.

I created the release into SourceForge.Net and uploaded files there at Thursday. I did not sent any notifications about the release. I only sent e-mail to our development group mailing list asking people to download it and do some final testing over the weekend, before we do the real announcements. This worked fine with some older releases.

Why don’t we just announce it immediately? To protect users from faulty releases. Errors happen, and it is a lot better if we catch and fix errors before tens of thousands of people download the release. I thought this kind of “half-public” release would be a good way to offer release for limited amount of users to test.

Why it now failed is because BetaNews picked the release in couple of hours. Wow, they are really fast! I don’t really know how they did it, do they have a way to monitor our releases-page or something like that? In SourceForge.net releases there is a monitoring feature. Which allows us to sent release notifications for registered people. However I did not sent notification until Friday.

NOTE: I’m not saying BetaNews did something wrong. They did exactly what they are supposed to do – offer releases for users. It just surprised me they are so fast. This is my fault, nobody else’s.

And after BetaNews had the release, lots of blogs and other sites had the release. So here we are. I’m sorry about the confusion this may have caused.

Update 2008-04-06: Web site is now updated and release announcements done.