Behavior change in 2.12 – FAIL!?

One important behavior change in 2.12 release was WinMerge now includes all files in all unique subfolders to the compare. Previous versions just showed that there was an unique folder.

Old behavior had many problems, we really didn’t know (and hence could not show) what was contained in those unique folders. Users complained when they could not copy/delete just certain items from those unique folders. It prevented developing many new features as the result set was just part of the reality.

In fact one very reason I finally implemented this was I can implement filtering for file copying. Meaning we apply file filters when copying items. That means user can set filter to filter Subversion’s .svn -folders. And when user then copies folders WinMerge does not copy those .svn folders. This is the feature I personally miss most. And seems many users agree it would be good feature, some even submit bugs about it.

So I thought I finally got it right when I re-wrote code to have all items in all unique folders included in compare results. It was a change I had been planning to implement past 5 years. And actually I had submitted couple of patches earlier but those patches weren’t good enough to commit. So I was actually quite happy that I got it done now.

Of course there are always users that are unhappy when something changes. I was expecting that and didn’t react to some complaints other than simply explaining it is the current behavior. Until I read this message in the forum saying new feature causes user’s usual compare times to grow dramatically (from under a second to 5-10 minutes). This was quite a shocking message to read!

And what is worse, there are no good solutions to this. I’m not sure if there are even sensible solutions we could implement. No way we go back to the old buggy behavior.

The problem is not just WinMerge – it is a lot more problem of operating system and hardware. OS and hw have caches that store last used disk data to memory to prevent slow disk access. But with this kind of huge folder structures most of the data must be read from the disk. And what is worse with WinMerge, we have two different folders structures when we compare, not just one. OS and hw caches assume programs use small set of files whose data can be kept in the cache. But then comes WinMerge and compares 100 thousand of files from two folders… And caches give up.

At the moment I don’t have any realistic ideas for solutions. It might be we can optimize the unique folders case somehow. Or delay reading the data, reading data in background etc. It is sure we need to do something…

Software is sometimes so fun!

First usable 2.13.x experimental release?

Few hours ago I released 2.13.4 experimental release. I hope this one is finally the first usable 2.13.x experimental release. Previous 2.13.x releases have each had some embarrassing (for me!) bugs. Of course I’m eating my own dog food and been using these releases. They are almost usable if one knows how to avoid those bugs and buggy features.

Perhaps the Most Embarrassing Bug I’ve been doing for a long time happened with 2.13.1. #2515875 “File->New” creates “disappeared” message in WinMerge 2.13.1. Funny (not really!) thing was that this bug was caused by my fix to another bug:  #2493945 Crash if one of the diffed file was erased by another applic. And that bug happened because my (thousand times checked) saving code didn’t check if the file really existed… Do you see the pattern? From rare crash bug we ended up to making  one of the most important features unusable. I’m always so proud when couple of hours after the release bug reports hit my inbox…

Sometimes I’m not sure which is more embarrassing, the fix for stupid bugs like these, or “thoroughly” tested fix that has funny side-effects.

The crash bug itself has been surviving few million users without reports until now. Which always begs the question if it really doesn’t happen for users or if they just don’t bother reporting it (as it is so rare). In this case I really hope it is the latter. As it is pretty easy to figure out situations when this happens. The simplest one being comparing files from removable media and then removing the media.

The only good thing is that my co-workers don’t see these bugs and mistakes… Oh, wait…

ToDoList

My todo-list is..huge. For WinMerge and Frhed there are hundreds of bugs to look and and (some day) fix. And few times more of new features and improvements (visible ones). And then maybe couple of thousand things for testing, documenting, refactoring code…

Few years ago this almost drive me nuts even trying to somehow keep up all these things. Not alone trying to track these things. Planning is totally out of the question. Most of the time I try to find the most important things. Based on my memory and experience about what are important things. Yes, I simply forget lots of things. Just reading through most bug reports would take days of my free time. I haven’t even tried to read through feature requests in last couple of years.

Sourceforge.net does not really help software developers. Their trackers are plain awful to use. They are slightly better than putting TODO and BUGS -text files to Subversion and just editing those files. But not much. Recently they added Trac to their software offering. I thought we could finally start using decent tracking software. But there is a catch. You can export data from Sourceforge.net trackers. But there is no way to import it to their Trac installation. So no switch for us.

Couple of days ago I updated to the latest version of the ToDoList by dan.g. It is an excellent advanced todo-list application. I’ve been using it for years tracking everything. Its weakness is it is a quite complicated-looking. Luckily it can be configured, but it takes some time to find a configuration that works best for different needs. And even then it takes more time to find out the best way to work with it.

Yes, there are lots of todo-list applications. Everybody wants to do their own. Just because they can? But most of those applications can only sort by few levels of priority. Some allow even setting due dates. How original. I quess people writing those applications never have more than ten items in their lists. And they never want to categorize or sort their items. No much sorting in two items… When you have even 50 items you really appreciate ability to add categories etc to your items.

One nice thing in ToDoList is its code is available under free license. Few years ago I even borrowed some code from it to WinMerge (the options dialog). Unfortunately the current license (The Creative Commons Attribution-ShareAlike 2.5 License) is not GPLv2 compatible and hence does not allow further code sharing. I suspect the licensing change is pure accident due to code published as a part of the article. But still prevents me updating to latest options-dialog code. Of course I could ask if Dan would be willing to double-license the code with GPLv2 compatible license…

Small annoyance with ToDoList is it doesn’t have installer but just files in zips. So the update is not simply downloading installer exe and running it (which I’ve used to doing for lots of programs). Instead I need to unzip the files with WinRar. That gets annoying when there are frequent updates.

If you need a good todo-list but don’t want to install Bugzilla for it, ToDoList is worth looking at.

3500 commits (and six years)

Yesterday I noticed that my commit count in ohloh.net has reached 3500 commits. It is the amount of commits I’ve done to public Open Source repositories that ohloh.net tracks. It took about six years.

Now, that number is something one can proud of and it is public record. It is not the whole truth however. I don’t have easy way to prove it but I think my real commit count is over 4000. The reason? Ohloh.net tracks only commits to SVN trunk. So all branch commits are missing. We have sevenrelease branches in WinMerge and every release branch easily gets fifty or more commits by me. So 500 commits to branches sounds realistic. But as the 3500 is easy to prove and clearly visible it is better number to use.

Another unfair point with this number is that all commits are not for my work. I’ve just committed lots of work done by others. Translation updates are one example where I just commit work of others. I always give the credit in commit log (and in contributors listings) but again there is no easy way to tell how many my own patches I’ve committed.  However, I do some work for every commit. I build, test, review etc. So I think it is fair to say all these commits are my work too.

How about this another number, six years? When I contributed first patches for WinMerge I merely did it so that I could give back something. I never intended to become a regular developer. Indeed I did not even care if my patches were accepted or not. Of course it was nice they got accepted quickly. And the last thing I needed back then was taking responsibility from a software like WinMerge. I was using some Open Source software but knew very little about development models, tools and so on. It was totally new world and development model for me.

All that learning has been the best side of all this Open Source work. You must learn something new almost every day. It has been a big benefit in my day work too. Many times I’ve noticed at work that I already know the tools and libraries I need to use.

Everything I do is public. Every good idea and clever piece of code (when I some day write one…). But errors too! It is not nice when people see all errors and mistakes. There is no way to hide anything once it gets added to public repository. It was a bit shocking at first. Sometimes it is quite embarassing to find out that bug is caused by my own commit couple of years earlier. But I’ve never really seen finger pointing in WinMerge. Its not important who did it (error). It is important we fix it. This is one thing I really like in current development team.

It has been a fun six years with WinMerge. And although I need small breaks now and then I’ve no plans to stop contributing. There is so much work to do. WinMerge is good, but it could be so much better… For Frhed there are lot more serious alternatives, but I think it has its own good sides. Maybe it won’t be the best free hexaeditor but it will be one good hexeditor with some unique features.

Version numbering

Version numbers are always interesting topic. Although they really should not be. But people and users have used to things like “bigger version number always means better software”. Which is a bit uncomfortable for projects like WinMerge.

After all the version number is an identifier for the release. We could just give names for our releases (like some projects do). Problem with names is there is no clear clue about what is the latest release. And WinMerge project releases pretty often, several stable releases within a year.  So version numbers have their merit too.

We have been actively releasing experimental releases which naturally have bigger version number than latest stable releases. We’ve named them experimental releases, we’ve written warnings in release notes that they really are not stable release. Still people use them because they have bigger version  number. And even many download sites offer our experimental releases because the bigger version number. And it gets worse – I tried to add latest stable release for one download site but could not add a release with smaller version number than some experimental release!

We are changing WinMerge version numbering a bit. It should help with understanding our version numbers. Yes, current numbering sounded nice idea few years ago but it has became quite a pain. Back then we thought about “advanced” users who could figure out our system. But it has became painfully clear that many users don’t (bother) figuring it out but just download the version with biggest version number. No matter about categories or descriptions. Users want to use the latest and greatest version. Without thinking our nice numbering ideas.

It is hard to arrange releases so that the latest stable release will have the biggest version number. Only solution I could think of is using totally different numbering for unstable and stable  releases. But then the relation and order of releases would not be clear anymore.  I still want that we have linear release history so that certain unstable release series leads to stable release series. Unstable release series adds features and fixes for the next stable series. It is continuous development, not random hacking.

I personally feel it confusing that some projects use same numbering for stable and unstable releases. Sometimes I need to reads some release notes and changelogs to know if the release is stable or unstable as the version number and/or name does not tell that vital information. There are projects I don’t care being tester myself but use the stable releases. So WinMerge (and Frhed) will have clearly separated version numbers for stable and unstable releases.

In 2.13.x and later versions we only have three version numbers. First is two always. We aren’t ready to release 3.0 version of WinMerge. Actually I wonder when we could be. But updating version number to 3.0 would mean for users there is some real big changes. And I doubt we ever have releases with “big” changes. So we’ll stick with 2 for now on. Perhaps things like real cross-platform support or real 3-way merge could make the 3.0 release…

Second number is even (2.12) for stable releases and odd (2.13) for beta- and experimental releases.  Third number is just counter. It starts from 1 and we’ll increase it by one for each experimental release. For beta release we may bump it a bit more. For example first 2.13 experimental release was 2.13.1. Next will be 2.13.2 and so on. First 2.13 beta release will probably be 2.13.10 or 2.13.20, depending how many experimental releases we release before it.

This also means making the difference of beta- and experimental-releases smaller. They both are unstable releases from user’s point of view.  And we may in future just combine these two categories into one “unstable” category. Simplicity is good.

Frhed 1.3.10 Beta

Couple of days ago I did the Frhed 1.3.10 Beta release. This beta release is the last of 1.3.x releases before the 1.4.o stable release. Expect the stable release in about two weeks.

The start of the Frhed development has been very very interesting. The codebase we started working was good, but old. There weren’t many bugs, but the code, GUI etc was designed for Windows 9x. So the most important thing so far has been to make sure Frhed works correctly in latest Windows versions. And update the GUI to match current practices and guidelines. We’ve advanced a lot, but a lot remains to be done after 1.4. release too.

Download counts of Frhed alpha releases have positively surprised me. I expected to get few dozens of downloads. After all it was an alpha release. Intended for testing, not for production use. But there are already over 4700 downloads. So people have already found the tool. It will be interesting to see how much downloads the stable release gets later.

Frhed has been also very refreshing project for me personally. As it is in very early stages (from my point of view we just have started with it). We can just rewrote or remove things we don’t like. No need to think about hundreds of thousands of users. It is sometimes quite hard to improve things in WinMerge since bigger changes always affect to users. And not always in positive way. With Frhed we just don’t care yet.

For Frhed we start really caring perhaps after couple of stable releases. When we have some idea what users want and need from the software. Do they want simple and fast hex editor? Or do they want more features rather than simplicity and speed? We’ll find out but it takes some time. Then we must choose our own way to go.

Frhed 1.3.4

Today I made the Frhed 1.3.4 alpha release.  As I’ve outlined in my forum post, I expect to have Frhed 1.4 stable release in about month.

For that target this 1.3.4 alpha is the most important release. It already is pretty close to stable release in features. Maybe we’ll add some new features, but nothing major. Most changes will be smaller fixes and improvements. Manual is one part I’d like to improve but lets see if I find time…

Previous Frhed releases were more of setting the infrastructure (dowloads, web page, trackers, etc etc). Frhed 1.3.3 already had lots of things in place but missed many required bug fixes and improvements. So in my book Frhed 1.3.4 is also the first real Frhed release. And I like it!

Please test and use it. If you find bugs, please report them to our bug tracker. If you have ideas for improvements, submit them to our feature request tracker.

Configuring WinMerge for Mercurial

In earlier post I wrote about WinMerge and TortoiseHG. Unfortunately there is nothing new to tell about that. Mainly because I’ve been busy with many other things. But I’ll update the Todo item #2190215 TortoiseHG integration when there is something happening.

Meanwhile, there are instructions in Mercurial wiki about configuring WinMerge to work with Mercurial. The instructions assume WinMerge is in $PATH. After the installer this isn’t unfortunately true. Feature request for it is #2029425 Installer: option to add WinMerge to PATH and I’ve just submitted a patch: #2435175 Installer: Add WinMerge to system path.

The important part in the wiki page is how and what paths to give to WinMerge. This line describes it:

winmergeu.args=/e /ub /dl other /dr local $other $local $output

which looks lengthy so I’ll explain it briefly:

1. -e – tells WinMerge to close with single ESC key press
2. /ub – disables adding paths to MRU list in Open-dialog
3. /dl other - gives left-side file a label ‘other’
4. /dr local – gives right-side file a label ‘local’
5. $other and $local are paths to open
6. $output is the path where the merged file is written to

More about WinMerge command line in Online manual.

This command line format is what I’ve always looking for when I’m integrating WinMerge with some program. Now that the correct command line is known (to work) it is a lot easier to figure out rest of the pieces.

And if I understood tksoh’s comment in Todo item #2190215 correctly then we only need to add these settings for Mercurial, not for TortoiseHG.

Case collision in Frhed repository

I just hate that Windows doesn’t care about case in file/folder names. Today I hate it very very much. I’ve spent hours and hours trying to get working Mercurial repository converted from Subversion repository.

It should be trivial thing to do. I’ve converted many other repositories without problems. And indeed I’ve been happily using the import I did couple of months ago. Then at Monday when I was traveling without network access I decided to do something I hadn’t do before with this repository: I wanted to do a new clone and develop with it. I expected it just works as Mercurial usually does. BUT cloning aborted with error message “abort: case-folding collision between FRHED/frhed.dsp and FRHED/FRHED.dsp“. WHAT??

Digging into history I noticed there is commit by Jochen (SVN rev 46) which renames FRHED/FRHED.dsp to FRHED/frhed.dsp. That looks so innocent change and indeed should not cause problems. But for some reason I haven’t yet figured out the FRHED.dsp is not really removed from the converted Mercurial repository. So now when I try to update the working copy (clone, update etc) there are two similar filenames (as Windows sees them).

I expect this is two-fold problem. First the Subversion doesn’t do real renames for the files, but it does copy + delete for the file. And then (I suspect) hgsvn doesn’t detect the rename correctly as the filename doesn’t change (from Windows point of view).

There is a page in Mercurial wiki about fixing case collisions. I’ve tried many variations of that. It basically works as it creates repository I can clone. But as the “fixing” commit is naturally the last commit in the repository it doesn’t help with the problem unable to update working copy to earlier revision. I’m afraid I have to live with this restriction.

Out of interest I’m also trying Mercurial Queues to edit the history and remove the FRHED.dsp file after the rename. In hope I can get a fully working repository.

Yes, I know. Editing the version control history is bad idea. You simply should not do it. Especially with distributed version control system like Mercurial. Mercurial wiki has a good page about this.

But I want to experiment this (I’m sure I learn a lot about Mercurial in the course). So far it seems it can be done, but it requires hand-editing many revisions. And indeed I just realized I tried to do it wrong way around.. Whoops.

The principle here is MQ exports the history as set of patches. Then the patches can be edited, reordered, removed etc. And applied back to the repository after the changes. My idea is to remove the commit that renamed the file (so the original FRHED.dsp filename remains untouched). And then edit further patches so that they edit FRHED.dsp file instead of frhed.dsp.

As I said above I already started doing this wrong way – I removed the rename commit and started editing patches so that they edit the frhed.dsp file instead of FRHED.dsp. But the problem is of course that if I remove the commit, there never will be frhed.dsp file… Doh.

After some more experiments I’m stuck with new problem. Edited patches don’t apply cleanly and at the moment I’m still wondering why. I mean, the patches don’t apply for files whose diffs I don’t edit in patch files.  If I just remove hunks for editing FRHED.dsp why hunks for heksedit.dsp fail… This is a bit weird. But I’m new with this MQ stuff so perhaps I just did something wrong. I need to learn more about MQ.

But I’ll leave that for tomorrow. I’m still positive I can get working repository out of this.

Update: After several attempts I finally managed to fix the Mercurial repository.

Things got wrong in SVN rev 46 (as mentioned above). The hgsvn conversion script records rename of FRHED/FRHED.dsp to FRHED/frhed.dsp as copying of FRHED/FRHED.dsp to FRHED/frhed.dsp. So it won’t remove original file, just creates a new copy with new name. Which only differs by case and causes Windows to not see it as two different files.

And then things go totally wrong in conversion. The script generated diffs that made same changes to both FRHED/FRHED.dsp and FRHED/frhed.dsp files. Which in addition to all other problems also caused the patches to fail to apply by hand (as the first change already applied the change and second was then invalid).

Anyway, the working solution I found was to convert the faulty revision to real rename. I did this with MQ (see above). I reverted the revision 46 (SVN) and replaced it with real (Mercurial) rename of FRHED/FRHED.dsp to FRHED/frhed.dsp. Next problem was there were set of patches referring to now inexisting file. Simple solution was to do a search in files for FRHED.dsp for the patch files and remove all changes to that file. Once I had fixed the patch set I simply re-applied it to repository.

And the end result is the FRHED.dsp file disappeared. And I have exactly the repository I wanted.

I’m not sure how to avoid this kind of problem in the future. Best would be to avoid this kind of renames. But if somebody does the rename then there is no way to really backout it from Subversion repository. So sometimes manual fixing is needed. Luckily I now know how to do it in few minutes.

The exact procedure is:

1. convert the SVN repository to Mercurial using hgsvn
2. CD to Mercurial repository
3. initialize MQ: hg qinit
4. export history to patches, including the faulty revision: hg qimport -r FAULTY:tip
5. apply the faulty patch: hg qpop - copies FRHED/FRHED.dsp as FRHED/frhed.dsp, but nothing happens in the filesystem
6. fix the result: hg mv FRHED/FRHED.dsp FRHED/temp.dsp; hg mv FRHED/temp.dsp FRHED/frhed.dsp – this results in Mercurial seeing real rename. We must use temp as filename as intermediate step.
7. tell MQ that the patch was modified: hg qrefresh
8. edit patch set in .hg/patches folder and remove references from all patches to FRHED/FRHED.dsp file
9. apply the fixed patch set: hg qpush -a
10. delete the patch queue: hg qdelete -r qbase:qtip
11. Now we have fixed Mercurial repository

New Frhed 1.3.2 alpha release

Yesterday I made second release in two days. Luckily Frhed releases are still a lot less “major” releases than WinMerge stable releases are.

Frhed 1.3.2 alpha release is still setting the basics for the project. There is no user-visible changes (other than couple of new separators in File-menu) compared to earlier releases. But this releases installs important documents like license text and contributors list with other documentation. Also for now on I’ll be using Visual Studio 2003 instead of VC6 for compiling. Hence the runtime files for VS2003 were added to install.

Switching the compiler is not always easy. But For Frhed it was pretty painless. And I have quite a lot of experience from VC6 to VS2003 update from WinMerge project…

There is a lot of code cleanups which is kind of base job for further improvements. I also finally got the release script working so further releases are easier and faster to do.