Two Directory Source Control Workflow

Two Directory Source Control Workflow

Two Directory Source Control is a work flow where you utilize one directory from source control, one directory that you work out of, and you then shuttle between them at your convenience. In this article, I’ll describe how traditional work flows work, the problems with them, and how you can use the Two Directory option to overcome them. This article assumes you know what source control is, what a merging tool is, and the value they bring to your work flow.

Traditional Source Control Work Flows

Many tools such as Flex Builder , Flash , and Visual Studio have plug ins that allow integration with source control such as Subversion , CVS , Perforce , and Visual Source Safe . This allows you to check out code & assets, edit them, and check your changes back into source control directly from your IDE of choice. Tools such as Subclipse , a plug in for Eclipse , allow you to check out & check in code from an SVN repo directly into your project. It also has nice built-in "diffing", the ability to show 2 pieces of code side by side to allow you to see what’s changed, and merge them if you so desire.

The reality, however, is that these work flows aren’t as easy as the marketing makes it sound. The main flaw they all have is conflict resolution.

Conflict Resolution

Conflicts are when the code you have on your machine is different from what is in source control. Whether this is because the file in source control is newer, someone has the file checked out already in Visual Source Safe, or a folder/package structure has been re-arranged making your structure out of sync. My main gripe with the tools I’ve used is that they do not do an adequate job of making conflict resolution easy. I define easy as simple to understand what the conflict is, how to resolve it, and not interrupting your current work forcing you to immediately deal with the conflict. To expound on that last part, I agree that you must deal with conflicts when checking in code, but a conflict should not immediately break your current build until you resolve a conflict. Nor should it prevent you from working.

There are 2 scenarios where this irritation is shown. The first example is where you check out the latest code from SVN for your project. You then make your Flex Project in that directory. Assuming you’ve figured out how to prevent Tortoise SVN from copying .svn folders into your bin directories, you download the latest, and get to work. How long you work, however, is based on how often your team checks in code. Even if you are working on different portions of the code, if someone checks in code that relates to yours, you’ll have to deal with the conflict. Typically, you’ll do an update first, and once you’ve ensure none of the code you are committing needs to be merged via resolved conflicts, you commit your changes.

As your team size grows, however, many people will be checking in code. Since dealing with conflicts isn’t fun, sometimes teams will check in more often. This results in a behavior re-enforcement for others to check in sooner as well. While I’m a firm believer in having something compiling sooner than later is a great Agile practice to follow to reduce risk, it’s hard to really make head way on some of your code if your focused on getting code compiling quicker so you can check in vs. making progress on your code.

It may seem subtle, but it’s a major attitude adjustment that I perceive as harmful for developers to have. They should enjoy coding their masterpieces, and assuming they are communicating with others and/or their team lead, they should be focusing. A developers focus is a very important commodity that shouldn’t be taken lightly. Instead of coding by positive reinforcement, "I can’t wait to check this in for others to see!", they are instead coding by negative reinforcement, "I need to hurry up and check in what I have working so I don’t have to deal with a confusing merge operationthat’ll slow me and my team down".

Eclipse combined with Subclipse has some good merging and history management tools. Eclipse has it’s own local history where it stores multiple versions of your files. You can choose to revert if you need to and it’ll provide a diffing tool right there for you. My first problem is the diffing tool sucks. Subclipse is not much better. I last used it in February of 2007; if it has improved since then, let me know. This wouldn’t be a big deal, except when it’s 5:59 pm on a Friday evening, and you really really want to go home. Unless you resolve the conflict, your build will be broken, and you cannot commit till you do so. Suddenly, a good diff tool becomes very important. It’s really frustrating for a developer, and removes motivation to work later. To compound it, most developers I know refuse to leave until their code is compiling. Suddenly an update interferes with this.

The second scenario which is worse than the above is package re-factoring. This is when you change the directory of a file, or a whole folder structure. If it is minute enough, you can usually get by if its not a directory you were working in. If, however, it is AND files have change, this can be a really long and frustrating process… all the while your code no longer works. You cannot compile JUST because you wanted to do the right thing and check your code in. This is also where you can truly tell if a diffing tool is good or not; how it helps you resolve folder changes and their relations to files.

One solution to this is to check your code into a branch and merge in later when it is convenient for you.

The last problem with working directly from the repo is actually caused by a feature of a lot of source control applications. This is the ability to only check in a directory with files that you are working on. For example, if your team is using Cairngorm, and all you are doing isValueObject and Factory work, you can pretty much stay in those 2 directories, and not worry about someone stepping on your toes.

…it is a false sense of security, however. The cardinal rule for source control is only check in code that compiles. This doesn’t mean that it compiles in your machine. What this means is if you were to check out the entire build from source control, could you get to compile? By updating just your area of interest from source control, and then committing your changes successfully in that area, in this case "vo" and "factories" packages, does not mean you have followed this rule. If another developer working on the Cairngorm "delegates" was using your Factories, her/his code would break even though you successfully checked code in. In effect, you broke the build. At one of my older jobs, whoever broke the build had to change their IM icon to a monkey. It sounds trivial, but it really sucked when people asked you why your IM icon was a monkey…

To prevent this, you need to update the entire repo. For some repo’s, this is not easy to do. Then one I’m working with currently is at least over 12 gigs, with hundreds of thousands of files, both ASCII and binary. That means that if at least 12 developers are working on it at the same time, chances are high a lot of files will be added and changed. Regardless, you only take the hit in the beginning, if you update frequently, and have a good diff tool, this increases your ability to ensure your code won’t break the build when you check in. Enterprise tools installed on the source control server itself like Cruise Control can actually automate this process, and send emails with information on who checked in code, if it compiled or not, and what was changed. To be fair, this isn’t a problem with a lot of independent server-side code.

Another minor point is how some source control systems work with locking files. Some allow developers to check out code and "lock" it. This prevents others from checking it in. Some make it impossible to work with because the code then becomes read-only. Since having another directory under your control, away from the source folder, you won’t have this problem.

To summarize, there are a lot of problems, even with built-in IDE tools, to working directly in the repo:

  1. conflict resolution prevents you from compiling. Manager says, "Can I see the latest?" Dev replies, "No… I’m merging code."
  2. folder change conflict resolution. "Can you re-factor later?"
  3. pain avoidance code committing. "I’d better hurry up and check in what I have working so I don’t have to merge… to heck with focusing".
  4. not checking out the entire repo, committing code, and breaking the build. "Works on my box…"


Two Directory Source Control Work Flow

To reiterate, Two Directory Source Control is a work flow where you utilize one directory from source control, one directory that you work out of, and you then shuttle & merge between them at your convenience. I’ve been using this work flow for about 2 1/2 years now and everyone I encounter usually finds it extremely odd I don’t work directly in the repo, hence the need for this blog entry. I was taught this work flow by Thomas Burleson, one of people responsible for the Universal Mind Cairngorm Extensions . Beyond the above, his additional justification at the time was that with Flex 1.5, SVN and Tomcat or JRun would do weird things together every so often, resulting in strange compilation bugs. Before Flex 2, you had to compile on the server (it was the law), so you’d setup a local web server on your box, and refresh a web browser page to see your app.

This work flow solves the problems above.

The first thing is conflict resolution is now done at your convenience. This is a big deal. Suddenly getting the latest from the repo is not a grand affair. You can just get the latest, and go back to work. You can also see what has changed without having to immediately integrate it into your code base if you’re not ready to do so. You can also keep working; since the code you updated is in a totally different directory, the code you’re working on isn’t affected. About to catch a plane an losing wireless? Just update, and you can still focus on coding instead of resolving your changes with someone else when you are incapable of giving them a phone call to discuss. Finally, you can confidence that when you update, your code will still work. That’s a good feeling.

Secondly, file conflict resolution is challenging, yet fun. Folder conflict resolution, however, is very challenging, and requires a lot of meticulous concentration. Deleting directories can delete a lot of files and folders within them, and thus tick a lot of other developers off if you screw up and check in your screw up. So you have to be careful. Rather than updating some massive package re-factoring and preparing for a battle with Subclipse, you can instead just let Tortoise SVN do it’s job; you just right click and update; done. When you’re ready, and you know your version of code works, you can then merge the 2 pieces using Beyond Compare, or whatever Merging tool you use. It’s a lot easier to do this when you have 2 real, physical folder directories vs. a virtual one.

Third, this convenience extends to code committing. Rather than being motivated to commit code out of fear of merging, you can do so when you’re good and ready. Even if you wait 3 days, you know you’ll have plenty of time to merge the changes into your code, get it working, THEN check in inside a safe, non-repo area. This confidence in your ability to merge with the assurance you have a nice sequestered area away from your repo motivates you to focus on creating good code vs. creating compiling code for the mere sake of making merging easier.

Fourth, because this work flow forces you to continually get the latest code form the entire repo, you always have the latest build, not just a portion of the build. This ensures that when you check in code, you know it won’t break the build because you already have the entire build and have tested it on your local box. Typically anything wrong from there is a configuration issue, not a code out of sync issue.

You need 2 things to utilize this work flow. The first is your folder structure and the second is a good merge / diffing tool.

Folder Structure

When setting up a project, you create your project folder. Inside of that, you make 2 directories called "dev" and "local". The "dev" folder is where your repo goes. This can be an SVN/CVS repo, or a Perforce work space. In the case of Subversion, I’ll usually have Tortoise check out trunk, tags, and branch here.

To say it another way, "dev" is where your source control code goes. The "local" folder is where your copy goes. You play and work inside of local. When you are ready to merge in changes, you use a merge tool to do so.

dev and local folders



Setting Up Beyond Compare

For the merge tool, I use Beyond Compare . While it’s only for PC, I purchased CrossOver , a Windows emulation tool for Mac, just so I could retain the same work flow on both my PC and Mac.

BeyondCompare has a few setup steps that you only have to do once. These are:

  1. Change default keyboard short cuts.
  2. Creating a new "session" to have your repo on the left, local code on the right.
  3. Showing only mismatches.
  4. Making your session expand folders & expand only folders with differences.
  5. Setting up BeyondCompare to filter (aka not show) certain files and folders
  6. Adjusting the file comparison control to be Size & CRC based
  7. Full Refresh!
  8. Synchronize to Right.


Let’s go over these steps in detail.

1. Change Default Keyboard Shortcuts

When you get the latest code, you’ll want to shuttle over to your local repository when you’re ready. If it’s new or unchanged files, you’ll want to do this quickly. I find Control + Right Arrow and Control + Left Arrow for putting local code todev is the best shortcuts for this.

Go to Tools > Options, and select "Keyboard" from the bottom left. Scroll down till you see "Copy to Right". Click it and press Control + Right Arrow key. Click "Copy to Left" and press Control + Left Arrow key.

BeyondCompare keyboard shortcuts

2. Creating a new "session" to have your repo on the left, local code on the right.

The easiest way to create a new session in BeyondCompare is to go "Session > New Empty". A session in BC is your saved settings with a name. A session will store what directory you are viewing on the left, what directory you are comparing to on the right, and other comparison options. You typically create 1 session per project. I typically put the dev, your source code repository on the left, and local, your copy, on the right.

Dev on the left, Local on the right

3. Showing only Mismatches

Click the "Only Mismatches" button. This will filter the files to only show files and folders that aren’t the same. Blue ones will be files that do not exist on the other side. Red indicates files that are different and gray indicates a filethat’s older than the file on the other side.

Show only mismatches

4. Making your session expand folders & expand only folders with differences

You want to have the views automatically expand folders if they are different. To do this, you need to check some check boxes in 2 places. The first place is in the Session settings under "Session > Session Manager". For your session, you’ll see a section called "Folder Handling". Check all 3 check boxes.

The second place is in "Session > Comparison Control". You’ll see the same 3 check boxes at the bottom of the dialogue; click all 3.

Expand folders automatically and only with differences

You’ll have to do this again in another dialogue; see step #6.


5. Setting up Beyond Compare to filter (aka not show) certain files and folders

There are some files you don’t care about. For Subversion, for example, these are ".svn". For Flex Builder, this includes .actionScriptProperties, .settings, etc. For Mac users who show all files, this is .DS_Store. You can filter these files out by clicking the glasses icon, and typing in the files and folders you want to filter out, 1 per line, and clicking OK.

While this will filter out files in the view, it WILL copy them if you shuttle folders back and forth. I never copy folders across inBeyond Compare , only files. The exception to this rule is if you do an "Actions > Synchronize Folders". This will obey your filter settings. If weird things start happening in your code, make sure you didn’t accidentally copy a folder with a bunch of .svn folders and files in it.

6. Adjusting the file comparison control to be Size & CRC based

To ensure you get a decent difference between files, choose both size & crc checking for file comparisons. You can choose this radio button in "Session > Comparison Control". See above picture.

modify file comparison and re-do folder expansion

7. Do a full refresh!

This will compare your entire folder & file structure. You can do so via "View > Full Refresh".

8. Synchronize to Right

If this is your first time shuttling files from your dev folder to your local folder, you can go "Actions > Synchronize Folders > Synchronize to Right".


Conclusions

That’s it, you’re ready to go! Get latest revision, do a full refresh, merge, code. Later, you can get latest revision, merge your code left, and check-in with confidence! In the end, Two Directory Source Control allows you to still compile your code when checking out the latest revision, makes it easier to resolve largere-factoring efforts, allows code committing to be a pleasurable endeavor, and ensuring you always check out the whole repo so you can confirm it builds on your machine with your latest changes. I hope it helps you out too.

Let me digress here a minute and point out how disappointed I am in the Mac platforms support of development tools. Tortoise SVN and Beyond Compare are fantastic tools for the PC. I have not found the equivalent for Mac. SvnX and the Tortoise equivalent for Mac really really suck, especially when you grew up in Tortoise SVN. Same thing on the Merge front; I have yet to find a good GUI based Merge tool for Mac. I’ll probably install Parallels on my new Mac, and stick to the PC version of Tortoise SVN & Beyond Compare since I still can’t shake Flash Develop .



11 Responses

  1. Do you dislike FileMerge? I’m no expert on merging tools, could you explain what the difference is?

  2. Thanks for the article. I have a need-hate relationship with version control. It’s usually not an issue when you are only working in a specific area and everybody else knows to stay away. But in order to do sweeping architecture changes and you almost have to tell everybody else on the project to take a vacation until you are done.

    I will definitely use this two directory method in the future.

  3. One interesting thing I would add to this is the usage of fla files in a repo. Ideally you want version control on those fla files, but svn doesn’t understand them. That means if you’ve even saved your fla (even without editing it), you get a conflict when checking it back in. The whole “lock” thing is worthless, because you end up having to revert an fla you may or may not have changed, and trying to “merge” two fla’s is an exercise in frustration to say the least.

    I haven’t found a solution for this but from what I’ve read, hope is on the horizon. Apparently the fla format will be turned into a zip format where the internal workings will be stored in xml and folders, similar to Mars for PDF. Hopefully that will allow a plugin to be written for svn that will allow the merging of fla’s.

  4. When dealing with projects that have multiple people, I’ll often use SVN “Branches” to achieve the same affect.

    Person A is going to be doing Development Task 1. PErson B is going to be doing Development Task 2. I’ll create a branch “DevTask1″ and a branch “Devtask2″ that each developer can work in. When they are done w/ development, I’ll merge those branches back into the trunk.

    Of course, this may get tedious if you’re dealing with a 12 gig 12 people project.

  5. In my experience, this works great for bigger projects, and horribly for short term Flash projects. I’m definitely a branch fan.

  6. huy

    Wouldn’t a distributed version control system like mercurial work better for what you describe here ? You can have a local repository where you can check in your own changes and then push your changes to the main server when ready.

  7. Very nice artice, the method you suggest here has a lot of similarity to the Git workflow:
    http://osteele.com/archives/2008/05/my-git-workflow

    Currently, we’re still a small team of developers, using SVN, and I experience the same things you described about checking-in and out from the repo.

    I guess that once you get used to thos kind of work-flow, life becomes easier, yet, I think what’s really needed is better refactoring tools to work on the local repository. Basically it shouldn’t work much different, it’s just that class rename for example, works only on linked resources inside your fb environment.

    You mentioned file moving and package name changes, IMO it’s a refactoring feature whose absence is unbearable…

  8. Great Article Jesse. Because of those specific issues with Flexbuilder and SVN, I’ve been doing the two folder deal for a few years now. I never told anyone though, cause I thought they would think I was crazy for doing it that way. ;)

  9. Peter

    Just imagine that your second directory is it’s own local repository, and you can merge between the two of them! Git does all this for you and more… You may also want to check out Bazaar. It’s a little more SVN-like, but I prefer git because of github.com

  10. Thanks Jesse,
    Very nice article on svn issues and Beyond Compare tool.

  11. Hey there Jesse-

    First off, let me say I’m sorry to hear about your medical ordeal! Life throws you some curve balls sometimes, huh? Anywho, I hope you’re feeling better and you’re getting your mojo back. Respect…

    So, first off, your comments about Eclipse’s diff tool are spot on. Your comments about Subclipse are true too… IF and ONLY IF… you are using the latest version as of Feb. 2007.

    Oh my how Subclipse has changed… First off there have been several minor revision updates to Subclipse since Feb. of 2007. The latest version for Eclipse 3.2+ as of the time of this writing is Subclipse v1.4.2 and let me tell you, it is a FAR cry from v1.0.5 (which is the version you had at best) mentioned in your post. In addition to overall stability and reliability, the new version has several new enhancements. The top three IMHO are as follows:

    * JavaHL 1.5.1 binaries included for Windows

    From the Subclipse website, ‘JavaHL is the name of the Java language binding provided by the Subversion project. JavaHL is an official part of the Subversion project, not the Subclipse project.’ I’m assuming (because I’m on a Mac) that these binaries are written and optimized specifically for Windows systems. This should mean that Windows users will have a major performance increase when accessing SVN repositories.

    * Latest SVNKit Beta release included

    The SVNKit Beta release is pretty stable. I’ve been using previous Beta versions, albeit on OS X, and the thing is sound as a pound. For the newly initiated, SVNKit is a Java package allowing Windows clients (which we all know don’t have a standard SSH implementation) to access SVN repos. via the svn+ssh protocol. This eliminates the need to install something like puTTY or similar to establish an SSH tunnel to the repository. This has been a HUGE deterrent in the past for Subclipse but I believe this is now a thing of the past.

    * Subclipse’s diff tool

    This thing is very nice indeed. It’s far and away superior to the diff tool of previous versions. In Eclipse 3.2+ when doing a compare with Subclipse, the diff tool will preserver color coding and formatting of your respective files. So, HTML, MXML, XML, JS or whatever will have the same look as it would if you were editing it (disclaimer: most of the hotkeys/shortcuts do not function in the diff tool. This is a minor gripe, but it would RULE if ctrl/command + D et al. worked in the diff tool)

    For those of you who use SVN (and that should be ALL of you ;) ) the update site is here:

    http://subclipse.tigris.org/update_1.4.x

    I promise you won’t be disappointed.

    TD