Learning more about Git…

I’ve been using Git for couple of years more or less frequently. Mostly I’ve been using git when developing Cppcheck. In past months I’ve also used git at work. Some time ago I’ve finally looked at really using git for my OSS development work with Cppcheck.

So far my git use with Cppcheck (and other projects) has been very simple. Just pulling latest changes to master branch, committing and pushing commits to same master branch. Cppcheck development does not use feature branches, just the master branch. And my contributions have been pretty small and straightforward touching code that others haven’t been touching (much). As I’ve mostly worked with Windows support and GUI.

Then I decided that I finally want to improve cppcheck Windows integration. It is barely working but a bit messy and there is a lot to improve. Similarly I wanted to look at really improving GUI.

For that I didn’t want to work in master branch. I didn’t even want to work on Dan’s tree because I would be experimenting a lot. Yet I want to share my code for others to comment and discuss. I was curious if git can do all this for me?

Luckily there is GitHub! I can make my own clone of repository with single button click. Then I can work on my own clone as if I was working on Dan’s tree. I can even add branches from Dan’s tree as branches to my own repository. So I can easily follow also the upstream development and merge upstream code to my changes. This opens lots of new interesting possibilities for sharing work and code.

Experimenting with some of the possibilities took couple of days and quite a many deletions of repositories and cloning them again. But it was definitely worth of it! I’m actually starting like the possibilities and flexibility of git! And I feel I’m only scratching the surface here…

I think git is power user tool. It requires quite a bit learning. It is quite frightening for people like me coming from simpler (and more restricted) tools. You can start using git pretty fast for simple things. And I guess most users are happy with these simple usage patterns. But if you really want to take advantage of all the possibilities of git you need to reserve time for learning and experimenting.

And if the simple use is all that is needed then the git might be the wrong tool. For example Mercurial and Bazaar are simpler and easier to use in many cases. And easier to learn.

Priorities

I always thought that I could arrange at least small amount of time for OSS projects. I’ve managed to do that for several years. Of course that has meant I’ve given up the time from other things. As I thought these couple of projects I’m working on are more important than some other things.

This thinking worked until about year ago. Then I simply decided to switch my priorities. I moved these OSS projects to bottom of my priority list. Giving almost no time for them. Which has shown pretty clearly in past months. I’ve put in total few hours in last 6 months for these projects. While earlier I could put same few hours in one day. And it unfortunately shows: both WinMerge and Frhed are practically stalled projects now.

Most important reason was I realized the work amount required by both projects is way past my current abilities. Both projects require some serious re-planning and rewriting. I could continue putting lots of my own time to both projects and have things improved in say next 12 months. I’ve written some forum posts along that thinking.

WinMerge development was fun for many years. I learned a lot about programming, releasing, documenting etc etc. It was also quite stressful. It is not easy to release tool which you know lots of users will rely their work. The amount of positive feedback that we got was definitely what kept me going and developing. And that positive feedback still comes in.

But the fact is I don’t simply want to do that anymore. I started developing WinMerge since I needed a good tool for myself. And my motivation has always been selfish – I needed good tool and I wanted to share my work for others. And exactly this motivation has disappeared – I don’t need WinMerge so much anymore and I’m quite happy using current versions already. So why bother putting lots of my time to something I don’t really need?

I feel I’ve done enough of this free work for free tools already. And its time to do something else. I’m not totally giving up developing OSS projects. I’m still randomly working on these projects (including cppcheck). But the time amount I spent on them is lot less than before.

Behavior change in 2.12 – FAIL!?

One important behavior change in 2.12 release was WinMerge now includes all files in all unique subfolders to the compare. Previous versions just showed that there was an unique folder.

Old behavior had many problems, we really didn’t know (and hence could not show) what was contained in those unique folders. Users complained when they could not copy/delete just certain items from those unique folders. It prevented developing many new features as the result set was just part of the reality.

In fact one very reason I finally implemented this was I can implement filtering for file copying. Meaning we apply file filters when copying items. That means user can set filter to filter Subversion’s .svn -folders. And when user then copies folders WinMerge does not copy those .svn folders. This is the feature I personally miss most. And seems many users agree it would be good feature, some even submit bugs about it.

So I thought I finally got it right when I re-wrote code to have all items in all unique folders included in compare results. It was a change I had been planning to implement past 5 years. And actually I had submitted couple of patches earlier but those patches weren’t good enough to commit. So I was actually quite happy that I got it done now.

Of course there are always users that are unhappy when something changes. I was expecting that and didn’t react to some complaints other than simply explaining it is the current behavior. Until I read this message in the forum saying new feature causes user’s usual compare times to grow dramatically (from under a second to 5-10 minutes). This was quite a shocking message to read!

And what is worse, there are no good solutions to this. I’m not sure if there are even sensible solutions we could implement. No way we go back to the old buggy behavior.

The problem is not just WinMerge – it is a lot more problem of operating system and hardware. OS and hw have caches that store last used disk data to memory to prevent slow disk access. But with this kind of huge folder structures most of the data must be read from the disk. And what is worse with WinMerge, we have two different folders structures when we compare, not just one. OS and hw caches assume programs use small set of files whose data can be kept in the cache. But then comes WinMerge and compares 100 thousand of files from two folders… And caches give up.

At the moment I don’t have any realistic ideas for solutions. It might be we can optimize the unique folders case somehow. Or delay reading the data, reading data in background etc. It is sure we need to do something…

Software is sometimes so fun!

First usable 2.13.x experimental release?

Few hours ago I released 2.13.4 experimental release. I hope this one is finally the first usable 2.13.x experimental release. Previous 2.13.x releases have each had some embarrassing (for me!) bugs. Of course I’m eating my own dog food and been using these releases. They are almost usable if one knows how to avoid those bugs and buggy features.

Perhaps the Most Embarrassing Bug I’ve been doing for a long time happened with 2.13.1. #2515875 “File->New” creates “disappeared” message in WinMerge 2.13.1. Funny (not really!) thing was that this bug was caused by my fix to another bug:  #2493945 Crash if one of the diffed file was erased by another applic. And that bug happened because my (thousand times checked) saving code didn’t check if the file really existed… Do you see the pattern? From rare crash bug we ended up to making  one of the most important features unusable. I’m always so proud when couple of hours after the release bug reports hit my inbox…

Sometimes I’m not sure which is more embarrassing, the fix for stupid bugs like these, or “thoroughly” tested fix that has funny side-effects.

The crash bug itself has been surviving few million users without reports until now. Which always begs the question if it really doesn’t happen for users or if they just don’t bother reporting it (as it is so rare). In this case I really hope it is the latter. As it is pretty easy to figure out situations when this happens. The simplest one being comparing files from removable media and then removing the media.

The only good thing is that my co-workers don’t see these bugs and mistakes… Oh, wait…

ToDoList

My todo-list is..huge. For WinMerge and Frhed there are hundreds of bugs to look and and (some day) fix. And few times more of new features and improvements (visible ones). And then maybe couple of thousand things for testing, documenting, refactoring code…

Few years ago this almost drive me nuts even trying to somehow keep up all these things. Not alone trying to track these things. Planning is totally out of the question. Most of the time I try to find the most important things. Based on my memory and experience about what are important things. Yes, I simply forget lots of things. Just reading through most bug reports would take days of my free time. I haven’t even tried to read through feature requests in last couple of years.

Sourceforge.net does not really help software developers. Their trackers are plain awful to use. They are slightly better than putting TODO and BUGS -text files to Subversion and just editing those files. But not much. Recently they added Trac to their software offering. I thought we could finally start using decent tracking software. But there is a catch. You can export data from Sourceforge.net trackers. But there is no way to import it to their Trac installation. So no switch for us.

Couple of days ago I updated to the latest version of the ToDoList by dan.g. It is an excellent advanced todo-list application. I’ve been using it for years tracking everything. Its weakness is it is a quite complicated-looking. Luckily it can be configured, but it takes some time to find a configuration that works best for different needs. And even then it takes more time to find out the best way to work with it.

Yes, there are lots of todo-list applications. Everybody wants to do their own. Just because they can? But most of those applications can only sort by few levels of priority. Some allow even setting due dates. How original. I quess people writing those applications never have more than ten items in their lists. And they never want to categorize or sort their items. No much sorting in two items… When you have even 50 items you really appreciate ability to add categories etc to your items.

One nice thing in ToDoList is its code is available under free license. Few years ago I even borrowed some code from it to WinMerge (the options dialog). Unfortunately the current license (The Creative Commons Attribution-ShareAlike 2.5 License) is not GPLv2 compatible and hence does not allow further code sharing. I suspect the licensing change is pure accident due to code published as a part of the article. But still prevents me updating to latest options-dialog code. Of course I could ask if Dan would be willing to double-license the code with GPLv2 compatible license…

Small annoyance with ToDoList is it doesn’t have installer but just files in zips. So the update is not simply downloading installer exe and running it (which I’ve used to doing for lots of programs). Instead I need to unzip the files with WinRar. That gets annoying when there are frequent updates.

If you need a good todo-list but don’t want to install Bugzilla for it, ToDoList is worth looking at.

3500 commits (and six years)

Yesterday I noticed that my commit count in ohloh.net has reached 3500 commits. It is the amount of commits I’ve done to public Open Source repositories that ohloh.net tracks. It took about six years.

Now, that number is something one can proud of and it is public record. It is not the whole truth however. I don’t have easy way to prove it but I think my real commit count is over 4000. The reason? Ohloh.net tracks only commits to SVN trunk. So all branch commits are missing. We have sevenrelease branches in WinMerge and every release branch easily gets fifty or more commits by me. So 500 commits to branches sounds realistic. But as the 3500 is easy to prove and clearly visible it is better number to use.

Another unfair point with this number is that all commits are not for my work. I’ve just committed lots of work done by others. Translation updates are one example where I just commit work of others. I always give the credit in commit log (and in contributors listings) but again there is no easy way to tell how many my own patches I’ve committed.  However, I do some work for every commit. I build, test, review etc. So I think it is fair to say all these commits are my work too.

How about this another number, six years? When I contributed first patches for WinMerge I merely did it so that I could give back something. I never intended to become a regular developer. Indeed I did not even care if my patches were accepted or not. Of course it was nice they got accepted quickly. And the last thing I needed back then was taking responsibility from a software like WinMerge. I was using some Open Source software but knew very little about development models, tools and so on. It was totally new world and development model for me.

All that learning has been the best side of all this Open Source work. You must learn something new almost every day. It has been a big benefit in my day work too. Many times I’ve noticed at work that I already know the tools and libraries I need to use.

Everything I do is public. Every good idea and clever piece of code (when I some day write one…). But errors too! It is not nice when people see all errors and mistakes. There is no way to hide anything once it gets added to public repository. It was a bit shocking at first. Sometimes it is quite embarassing to find out that bug is caused by my own commit couple of years earlier. But I’ve never really seen finger pointing in WinMerge. Its not important who did it (error). It is important we fix it. This is one thing I really like in current development team.

It has been a fun six years with WinMerge. And although I need small breaks now and then I’ve no plans to stop contributing. There is so much work to do. WinMerge is good, but it could be so much better… For Frhed there are lot more serious alternatives, but I think it has its own good sides. Maybe it won’t be the best free hexaeditor but it will be one good hexeditor with some unique features.

Version numbering

Version numbers are always interesting topic. Although they really should not be. But people and users have used to things like “bigger version number always means better software”. Which is a bit uncomfortable for projects like WinMerge.

After all the version number is an identifier for the release. We could just give names for our releases (like some projects do). Problem with names is there is no clear clue about what is the latest release. And WinMerge project releases pretty often, several stable releases within a year.  So version numbers have their merit too.

We have been actively releasing experimental releases which naturally have bigger version number than latest stable releases. We’ve named them experimental releases, we’ve written warnings in release notes that they really are not stable release. Still people use them because they have bigger version  number. And even many download sites offer our experimental releases because the bigger version number. And it gets worse – I tried to add latest stable release for one download site but could not add a release with smaller version number than some experimental release!

We are changing WinMerge version numbering a bit. It should help with understanding our version numbers. Yes, current numbering sounded nice idea few years ago but it has became quite a pain. Back then we thought about “advanced” users who could figure out our system. But it has became painfully clear that many users don’t (bother) figuring it out but just download the version with biggest version number. No matter about categories or descriptions. Users want to use the latest and greatest version. Without thinking our nice numbering ideas.

It is hard to arrange releases so that the latest stable release will have the biggest version number. Only solution I could think of is using totally different numbering for unstable and stable  releases. But then the relation and order of releases would not be clear anymore.  I still want that we have linear release history so that certain unstable release series leads to stable release series. Unstable release series adds features and fixes for the next stable series. It is continuous development, not random hacking.

I personally feel it confusing that some projects use same numbering for stable and unstable releases. Sometimes I need to reads some release notes and changelogs to know if the release is stable or unstable as the version number and/or name does not tell that vital information. There are projects I don’t care being tester myself but use the stable releases. So WinMerge (and Frhed) will have clearly separated version numbers for stable and unstable releases.

In 2.13.x and later versions we only have three version numbers. First is two always. We aren’t ready to release 3.0 version of WinMerge. Actually I wonder when we could be. But updating version number to 3.0 would mean for users there is some real big changes. And I doubt we ever have releases with “big” changes. So we’ll stick with 2 for now on. Perhaps things like real cross-platform support or real 3-way merge could make the 3.0 release…

Second number is even (2.12) for stable releases and odd (2.13) for beta- and experimental releases.  Third number is just counter. It starts from 1 and we’ll increase it by one for each experimental release. For beta release we may bump it a bit more. For example first 2.13 experimental release was 2.13.1. Next will be 2.13.2 and so on. First 2.13 beta release will probably be 2.13.10 or 2.13.20, depending how many experimental releases we release before it.

This also means making the difference of beta- and experimental-releases smaller. They both are unstable releases from user’s point of view.  And we may in future just combine these two categories into one “unstable” category. Simplicity is good.

Frhed 1.3.10 Beta

Couple of days ago I did the Frhed 1.3.10 Beta release. This beta release is the last of 1.3.x releases before the 1.4.o stable release. Expect the stable release in about two weeks.

The start of the Frhed development has been very very interesting. The codebase we started working was good, but old. There weren’t many bugs, but the code, GUI etc was designed for Windows 9x. So the most important thing so far has been to make sure Frhed works correctly in latest Windows versions. And update the GUI to match current practices and guidelines. We’ve advanced a lot, but a lot remains to be done after 1.4. release too.

Download counts of Frhed alpha releases have positively surprised me. I expected to get few dozens of downloads. After all it was an alpha release. Intended for testing, not for production use. But there are already over 4700 downloads. So people have already found the tool. It will be interesting to see how much downloads the stable release gets later.

Frhed has been also very refreshing project for me personally. As it is in very early stages (from my point of view we just have started with it). We can just rewrote or remove things we don’t like. No need to think about hundreds of thousands of users. It is sometimes quite hard to improve things in WinMerge since bigger changes always affect to users. And not always in positive way. With Frhed we just don’t care yet.

For Frhed we start really caring perhaps after couple of stable releases. When we have some idea what users want and need from the software. Do they want simple and fast hex editor? Or do they want more features rather than simplicity and speed? We’ll find out but it takes some time. Then we must choose our own way to go.

Frhed 1.3.4

Today I made the Frhed 1.3.4 alpha release.  As I’ve outlined in my forum post, I expect to have Frhed 1.4 stable release in about month.

For that target this 1.3.4 alpha is the most important release. It already is pretty close to stable release in features. Maybe we’ll add some new features, but nothing major. Most changes will be smaller fixes and improvements. Manual is one part I’d like to improve but lets see if I find time…

Previous Frhed releases were more of setting the infrastructure (dowloads, web page, trackers, etc etc). Frhed 1.3.3 already had lots of things in place but missed many required bug fixes and improvements. So in my book Frhed 1.3.4 is also the first real Frhed release. And I like it!

Please test and use it. If you find bugs, please report them to our bug tracker. If you have ideas for improvements, submit them to our feature request tracker.

Configuring WinMerge for Mercurial

In earlier post I wrote about WinMerge and TortoiseHG. Unfortunately there is nothing new to tell about that. Mainly because I’ve been busy with many other things. But I’ll update the Todo item #2190215 TortoiseHG integration when there is something happening.

Meanwhile, there are instructions in Mercurial wiki about configuring WinMerge to work with Mercurial. The instructions assume WinMerge is in $PATH. After the installer this isn’t unfortunately true. Feature request for it is #2029425 Installer: option to add WinMerge to PATH and I’ve just submitted a patch: #2435175 Installer: Add WinMerge to system path.

The important part in the wiki page is how and what paths to give to WinMerge. This line describes it:

winmergeu.args=/e /ub /dl other /dr local $other $local $output

which looks lengthy so I’ll explain it briefly:

1. -e – tells WinMerge to close with single ESC key press
2. /ub – disables adding paths to MRU list in Open-dialog
3. /dl other - gives left-side file a label ‘other’
4. /dr local – gives right-side file a label ‘local’
5. $other and $local are paths to open
6. $output is the path where the merged file is written to

More about WinMerge command line in Online manual.

This command line format is what I’ve always looking for when I’m integrating WinMerge with some program. Now that the correct command line is known (to work) it is a lot easier to figure out rest of the pieces.

And if I understood tksoh’s comment in Todo item #2190215 correctly then we only need to add these settings for Mercurial, not for TortoiseHG.