Two Directory Source Control Workflow

Two Directory Source Control is a work flow where you utilize one directory from source control, one directory that you work out of, and you then shuttle between them at your convenience. In this article, I’ll describe how traditional work flows work, the problems with them, and how you can use the Two Directory option to overcome them. This article assumes you know what source control is, what a merging tool is, and the value they bring to your work flow.

Traditional Source Control Work Flows

Many tools such as Flex Builder , Flash , and Visual Studio have plug ins that allow integration with source control such as Subversion , CVS , Perforce , and Visual Source Safe . This allows you to check out code & assets, edit them, and check your changes back into source control directly from your IDE of choice. Tools such as Subclipse , a plug in for Eclipse , allow you to check out & check in code from an SVN repo directly into your project. It also has nice built-in "diffing", the ability to show 2 pieces of code side by side to allow you to see what’s changed, and merge them if you so desire.

The reality, however, is that these work flows aren’t as easy as the marketing makes it sound. The main flaw they all have is conflict resolution.

Conflict Resolution

Conflicts are when the code you have on your machine is different from what is in source control. Whether this is because the file in source control is newer, someone has the file checked out already in Visual Source Safe, or a folder/package structure has been re-arranged making your structure out of sync. My main gripe with the tools I’ve used is that they do not do an adequate job of making conflict resolution easy. I define easy as simple to understand what the conflict is, how to resolve it, and not interrupting your current work forcing you to immediately deal with the conflict. To expound on that last part, I agree that you must deal with conflicts when checking in code, but a conflict should not immediately break your current build until you resolve a conflict. Nor should it prevent you from working.

There are 2 scenarios where this irritation is shown. The first example is where you check out the latest code from SVN for your project. You then make your Flex Project in that directory. Assuming you’ve figured out how to prevent Tortoise SVN from copying .svn folders into your bin directories, you download the latest, and get to work. How long you work, however, is based on how often your team checks in code. Even if you are working on different portions of the code, if someone checks in code that relates to yours, you’ll have to deal with the conflict. Typically, you’ll do an update first, and once you’ve ensure none of the code you are committing needs to be merged via resolved conflicts, you commit your changes.

As your team size grows, however, many people will be checking in code. Since dealing with conflicts isn’t fun, sometimes teams will check in more often. This results in a behavior re-enforcement for others to check in sooner as well. While I’m a firm believer in having something compiling sooner than later is a great Agile practice to follow to reduce risk, it’s hard to really make head way on some of your code if your focused on getting code compiling quicker so you can check in vs. making progress on your code.

It may seem subtle, but it’s a major attitude adjustment that I perceive as harmful for developers to have. They should enjoy coding their masterpieces, and assuming they are communicating with others and/or their team lead, they should be focusing. A developers focus is a very important commodity that shouldn’t be taken lightly. Instead of coding by positive reinforcement, "I can’t wait to check this in for others to see!", they are instead coding by negative reinforcement, "I need to hurry up and check in what I have working so I don’t have to deal with a confusing merge operationthat’ll slow me and my team down".

Eclipse combined with Subclipse has some good merging and history management tools. Eclipse has it’s own local history where it stores multiple versions of your files. You can choose to revert if you need to and it’ll provide a diffing tool right there for you. My first problem is the diffing tool sucks. Subclipse is not much better. I last used it in February of 2007; if it has improved since then, let me know. This wouldn’t be a big deal, except when it’s 5:59 pm on a Friday evening, and you really really want to go home. Unless you resolve the conflict, your build will be broken, and you cannot commit till you do so. Suddenly, a good diff tool becomes very important. It’s really frustrating for a developer, and removes motivation to work later. To compound it, most developers I know refuse to leave until their code is compiling. Suddenly an update interferes with this.

The second scenario which is worse than the above is package re-factoring. This is when you change the directory of a file, or a whole folder structure. If it is minute enough, you can usually get by if its not a directory you were working in. If, however, it is AND files have change, this can be a really long and frustrating process… all the while your code no longer works. You cannot compile JUST because you wanted to do the right thing and check your code in. This is also where you can truly tell if a diffing tool is good or not; how it helps you resolve folder changes and their relations to files.

One solution to this is to check your code into a branch and merge in later when it is convenient for you.

The last problem with working directly from the repo is actually caused by a feature of a lot of source control applications. This is the ability to only check in a directory with files that you are working on. For example, if your team is using Cairngorm, and all you are doing isValueObject and Factory work, you can pretty much stay in those 2 directories, and not worry about someone stepping on your toes.

…it is a false sense of security, however. The cardinal rule for source control is only check in code that compiles. This doesn’t mean that it compiles in your machine. What this means is if you were to check out the entire build from source control, could you get to compile? By updating just your area of interest from source control, and then committing your changes successfully in that area, in this case "vo" and "factories" packages, does not mean you have followed this rule. If another developer working on the Cairngorm "delegates" was using your Factories, her/his code would break even though you successfully checked code in. In effect, you broke the build. At one of my older jobs, whoever broke the build had to change their IM icon to a monkey. It sounds trivial, but it really sucked when people asked you why your IM icon was a monkey…

To prevent this, you need to update the entire repo. For some repo’s, this is not easy to do. Then one I’m working with currently is at least over 12 gigs, with hundreds of thousands of files, both ASCII and binary. That means that if at least 12 developers are working on it at the same time, chances are high a lot of files will be added and changed. Regardless, you only take the hit in the beginning, if you update frequently, and have a good diff tool, this increases your ability to ensure your code won’t break the build when you check in. Enterprise tools installed on the source control server itself like Cruise Control can actually automate this process, and send emails with information on who checked in code, if it compiled or not, and what was changed. To be fair, this isn’t a problem with a lot of independent server-side code.

Another minor point is how some source control systems work with locking files. Some allow developers to check out code and "lock" it. This prevents others from checking it in. Some make it impossible to work with because the code then becomes read-only. Since having another directory under your control, away from the source folder, you won’t have this problem.

To summarize, there are a lot of problems, even with built-in IDE tools, to working directly in the repo:

1. conflict resolution prevents you from compiling. Manager says, "Can I see the latest?" Dev replies, "No… I’m merging code."
   
2. folder change conflict resolution. "Can you re-factor later?"
3. pain avoidance code committing. "I’d better hurry up and check in what I have working so I don’t have to merge… to heck with focusing".
4. not checking out the entire repo, committing code, and breaking the build. "Works on my box…"

Two Directory Source Control Work Flow

To reiterate, Two Directory Source Control is a work flow where you utilize one directory from source control, one directory that you work out of, and you then shuttle & merge between them at your convenience. I’ve been using this work flow for about 2 1/2 years now and everyone I encounter usually finds it extremely odd I don’t work directly in the repo, hence the need for this blog entry. I was taught this work flow by Thomas Burleson, one of people responsible for the Universal Mind Cairngorm Extensions . Beyond the above, his additional justification at the time was that with Flex 1.5, SVN and Tomcat or JRun would do weird things together every so often, resulting in strange compilation bugs. Before Flex 2, you had to compile on the server (it was the law), so you’d setup a local web server on your box, and refresh a web browser page to see your app.

This work flow solves the problems above.

The first thing is conflict resolution is now done at your convenience. This is a big deal. Suddenly getting the latest from the repo is not a grand affair. You can just get the latest, and go back to work. You can also see what has changed without having to immediately integrate it into your code base if you’re not ready to do so. You can also keep working; since the code you updated is in a totally different directory, the code you’re working on isn’t affected. About to catch a plane an losing wireless? Just update, and you can still focus on coding instead of resolving your changes with someone else when you are incapable of giving them a phone call to discuss. Finally, you can confidence that when you update, your code will still work. That’s a good feeling.

Secondly, file conflict resolution is challenging, yet fun. Folder conflict resolution, however, is very challenging, and requires a lot of meticulous concentration. Deleting directories can delete a lot of files and folders within them, and thus tick a lot of other developers off if you screw up and check in your screw up. So you have to be careful. Rather than updating some massive package re-factoring and preparing for a battle with Subclipse, you can instead just let Tortoise SVN do it’s job; you just right click and update; done. When you’re ready, and you know your version of code works, you can then merge the 2 pieces using Beyond Compare, or whatever Merging tool you use. It’s a lot easier to do this when you have 2 real, physical folder directories vs. a virtual one.

Third, this convenience extends to code committing. Rather than being motivated to commit code out of fear of merging, you can do so when you’re good and ready. Even if you wait 3 days, you know you’ll have plenty of time to merge the changes into your code, get it working, THEN check in inside a safe, non-repo area. This confidence in your ability to merge with the assurance you have a nice sequestered area away from your repo motivates you to focus on creating good code vs. creating compiling code for the mere sake of making merging easier.

Fourth, because this work flow forces you to continually get the latest code form the entire repo, you always have the latest build, not just a portion of the build. This ensures that when you check in code, you know it won’t break the build because you already have the entire build and have tested it on your local box. Typically anything wrong from there is a configuration issue, not a code out of sync issue.

You need 2 things to utilize this work flow. The first is your folder structure and the second is a good merge / diffing tool.

Folder Structure

When setting up a project, you create your project folder. Inside of that, you make 2 directories called "dev" and "local". The "dev" folder is where your repo goes. This can be an SVN/CVS repo, or a Perforce work space. In the case of Subversion, I’ll usually have Tortoise check out trunk, tags, and branch here.

To say it another way, "dev" is where your source control code goes. The "local" folder is where your copy goes. You play and work inside of local. When you are ready to merge in changes, you use a merge tool to do so.

Setting Up Beyond Compare

For the merge tool, I use Beyond Compare . While it’s only for PC, I purchased CrossOver , a Windows emulation tool for Mac, just so I could retain the same work flow on both my PC and Mac.

BeyondCompare has a few setup steps that you only have to do once. These are:

1. Change default keyboard short cuts.
   
2. Creating a new "session" to have your repo on the left, local code on the right.
3. Showing only mismatches.
4. Making your session expand folders & expand only folders with differences.
5. Setting up BeyondCompare to filter (aka not show) certain files and folders
6. Adjusting the file comparison control to be Size & CRC based
7. Full Refresh!
8. Synchronize to Right.
   

Let’s go over these steps in detail.

1. Change Default Keyboard Shortcuts

When you get the latest code, you’ll want to shuttle over to your local repository when you’re ready. If it’s new or unchanged files, you’ll want to do this quickly. I find Control + Right Arrow and Control + Left Arrow for putting local code todev is the best shortcuts for this.

Go to Tools > Options, and select "Keyboard" from the bottom left. Scroll down till you see "Copy to Right". Click it and press Control + Right Arrow key. Click "Copy to Left" and press Control + Left Arrow key.

2. Creating a new "session" to have your repo on the left, local code on the right.

The easiest way to create a new session in BeyondCompare is to go "Session > New Empty". A session in BC is your saved settings with a name. A session will store what directory you are viewing on the left, what directory you are comparing to on the right, and other comparison options. You typically create 1 session per project. I typically put the dev, your source code repository on the left, and local, your copy, on the right.

3. Showing only Mismatches

Click the "Only Mismatches" button. This will filter the files to only show files and folders that aren’t the same. Blue ones will be files that do not exist on the other side. Red indicates files that are different and gray indicates a filethat’s older than the file on the other side.

4. Making your session expand folders & expand only folders with differences

You want to have the views automatically expand folders if they are different. To do this, you need to check some check boxes in 2 places. The first place is in the Session settings under "Session > Session Manager". For your session, you’ll see a section called "Folder Handling". Check all 3 check boxes.

The second place is in "Session > Comparison Control". You’ll see the same 3 check boxes at the bottom of the dialogue; click all 3.

You’ll have to do this again in another dialogue; see step #6.

5. Setting up Beyond Compare to filter (aka not show) certain files and folders

There are some files you don’t care about. For Subversion, for example, these are ".svn". For Flex Builder, this includes .actionScriptProperties, .settings, etc. For Mac users who show all files, this is .DS_Store. You can filter these files out by clicking the glasses icon, and typing in the files and folders you want to filter out, 1 per line, and clicking OK.

While this will filter out files in the view, it WILL copy them if you shuttle folders back and forth. I never copy folders across inBeyond Compare , only files. The exception to this rule is if you do an "Actions > Synchronize Folders". This will obey your filter settings. If weird things start happening in your code, make sure you didn’t accidentally copy a folder with a bunch of .svn folders and files in it.

6. Adjusting the file comparison control to be Size & CRC based

To ensure you get a decent difference between files, choose both size & crc checking for file comparisons. You can choose this radio button in "Session > Comparison Control". See above picture.

7. Do a full refresh!

This will compare your entire folder & file structure. You can do so via "View > Full Refresh".

8. Synchronize to Right

If this is your first time shuttling files from your dev folder to your local folder, you can go "Actions > Synchronize Folders > Synchronize to Right".

Conclusions

That’s it, you’re ready to go! Get latest revision, do a full refresh, merge, code. Later, you can get latest revision, merge your code left, and check-in with confidence! In the end, Two Directory Source Control allows you to still compile your code when checking out the latest revision, makes it easier to resolve largere-factoring efforts, allows code committing to be a pleasurable endeavor, and ensuring you always check out the whole repo so you can confirm it builds on your machine with your latest changes. I hope it helps you out too.

Let me digress here a minute and point out how disappointed I am in the Mac platforms support of development tools. Tortoise SVN and Beyond Compare are fantastic tools for the PC. I have not found the equivalent for Mac. SvnX and the Tortoise equivalent for Mac really really suck, especially when you grew up in Tortoise SVN. Same thing on the Merge front; I have yet to find a good GUI based Merge tool for Mac. I’ll probably install Parallels on my new Mac, and stick to the PC version of Tortoise SVN & Beyond Compare since I still can’t shake Flash Develop .