Version control in the studio
/ by Toni
Orange is not a huge studio, but of course we still need proper technical setup for fluent workflow. By putting the six workstations and a fileserver in place, connected via a gigabit ethernet, the basic infrastructure was built during the first weeks. Pre-production began using a plain shared filesystem, but now as we entered production phase in the sense that scenes are put together for the animatic, we decided to use a versioning system instead.
A major reason is ensuring that no-one ever destroys work by others, which can happen easily with filesystems by simply saving an older version over a newer one made by someone else in the meantime. But our tech-savvy animation director Bassam figured that even advanced features like branching, i.e. making an alternative version of the movie-in-the-making, might be useful later for e.g. testing changes in the main characters.
We chose Subversion (svn), a popular CVS replacement, as it is friendly for basic operations such as renaming files. It will be interesting to see how well it works for blend files and other animation source data, such as large textures,and whether it fits in the way artists work individually and in collaboration.
Version control systems have been mostly made with software development in mind, and are commonly used by programmers, but nothing prevents using them for other kinds of data too. We do have experience of having blend files in cvs and svn repositories from open source
games, like from the Freedroid project where Basse has contributed, and from Soya-using Slune, Balazar and Overdrive (Soya3d can use Blender to open .blend files, so the cvs versions of the games get the models and character animations straight from blends). Everything has worked ok there, so we dared to make this decision..
With almost only binary data, we do not get the informative ‘diffs’, i.e. the information about changes made in a particular commit (save to the repository), that versioning systems give for text based software source files. But the commit messages, i.e. the info that the person who makes a change / adds a new file to the repository must write when committing, do still give info about what has changed where (as long as the guys keep up writing them properly! :)
I’m used to using the command line, but some of the others prefer GUI tools (and me too sometimes). On Linux, we are using both Gnome and KDE Desktop environments. For KDE there seems to be a nice plugin for the Konqueror browser / file manager, called ksvn, that I’ll try. On Gnome Bassam tried one Nautilus script which didn’t seem to work too well (it showed the svn commands also for files that were not in any svn working directory). But eSVN seems ok, so that’ll probably serve Lee who’s used to GUIs and using Gnome now (although eSVN uses qt if of course works in Gnome too).
On the macs, guys are now using a Finder plugin (which is open source) that works nicely for basic operations. For advanced operations, an open source macosx app called svnX seems great: when looking at the log, and selecting some commit/revision, the integrated browser shows that version of the project, and let’s the user open an old version of any file from there. I wonder if that, probably Cocoa-using, svnX tool would be easy to port to GNUStep..
We are still using the file share (samba) for non-versioned files, like reference photo collections, and also for saving rendered animations. The idea is to keep the production folder in the svn repository clean from extra data, but with all the necessary files for rendering, so that we can use Subversion also for synching with the render farm in the U.S. later on.
For the future, it might be interesting to investigate if blend files could be somehow ‘diffable’, although it may be that the range of operations done in 3d modelling and animation can not be captured in that way. Concerning using binary data, Subversion seems better fit than CVS, and reports even using binary diffing so conserving the full history might not take an awful lot of extra space. I haven’t looked how well it is actually able to diff e.g. blend files in this space-preserving sense, nor don’t know if they are suitable for such diffing at all.
So as it is, we seem to have a working solution, and the animatic progresses! *knocked-on-wood*
~Toni
« Licence to drive! (now with blend) | Hairy Issues // updated! »
Cool! On GUI clients for subversion, on Windows you can get TortoiseSVN (http://tortoisesvn.tigris.org)
Very interesting. I’ll try too the SVN (on Gnome, so probably i’ll try eSvn), it seems better then CVS and very useful to recreate the history files. Thanks for sharing your solutions!
;-)
I am also preparing a project similar to this one, it will start February next year and will last aproximatelly 11 months, since it will be mostly online. The team will only be together “live” for just about a month during the whole project.
The project is similar to Orange, but the movie will be done with several 3D applications (I will mostly use Blender) and rendered in Pixie (Renderman Compatible). Also with one big difference, the movie will be done entirely by Portuguese artists.
I am also targeting to use SVN for version control, but I will go further, we will do an application to manage the whole process behind the making of the animation/movie, using SVN as the backend we will build the whole frontend to manage the creative process, the development process and all other roles needed for this sort of production.
The application will be hosted at SourceForge, starting from February the 1st, it won’t be of much help for Orange, but if “Orange 2” will ever be done it will be as use then.
What I mean is, I know your residents team is short, but are you going to develop something to manage the whole process, or will you use different tools to do it?
Also, I would like to have as much feedback from you guys as possible, since I am going to be developing an OSS Application to manage a similar project.
Will meet with you at BlendConf … I will be the guy bugging you for feedback :)
Still on Kde, network operations are more transparents and completes in Konqueror, moreover, Kde is the best desktop eVer :)
Anyway, it’s interesting as approach for managing files in a movie production.
best regards!
Ton hinted at least weeks meeting of plans for an ‘Orange2’, so it looks like it is planned, but no official word,
LetterRip
For gnome you could use:
http://rapidsvn.tigris.org/
Gnome can’t bite correctly, its component gnome-vfs is so ugly…
Hey we did the same thing over at blenderprojects.com evertime you take a file, modify it, and re-upload it it produces a timestamp added to the name. This way we have a version for each stage of a model. Large files like the basescene are updated by the user working on that file.I believe we are on Basecene v7 right now. Soon to be v8 after today. This way if someone modifies a file and we dont like it we can go back 1 version and it is not lost. This has saved us many times already.
Are you aware of http://projects.edgewall.com/trac/ ? I integrated with subversion, some interest project tracking features and is written in python. Would be interesting if someone ehanced the python based diff functionality to leverage blender to produce a “.blend” diff either graphically or meta data display. Not sure if possible or useful, but would be an interesting approach for collaborative media projects. Maybe even evolve into a wiki engine for blender based media projects in similar spirit to video concept of http://www.annodex.net/. Just some ideas from an outsider… :-)
And let me re-iterate all the praise, Orange is an awsome project!! Blender is an awsome tool!!
Subversion is cool, but has some issues with large binary files. E.g. it tries to compress everything them before sending on the wire, which is pointless for most of the binary data (most binary files don’t compress that well anyway). But obviously SVN is still much better than CVS on that front.
If you want a really fast solution, try Perforce. I know that most game developers use that for art assets (big games – where data with file histories is terabytes), and it’s fast and scales well. I think it’s free for non-commercial projects (but you’d have to check that).
Frank, sure, I was very happy about trac in the previous project I was (and still am, hopefully doing Blender integration :), working on , the interactive Python notebooks.
In fact, installed trac on our internal svn server yesterday, but had some strange problem with apache config .. even though enabled cgi handling in all the ways that could find, it still shows the Python source, so gotta fix that at some point. Haven’t used apache that much anymore lately so am not too clear about all of it.
I actually did not know earlier that trac was in Python, but had assumed Perl or something, so it was a pleasent surprise. Exactly because there are all kinds of interesting possibilities for integration.. keep the ideas flowing! :)
NeARAZ, thanks for your remark on Subversions problems with large binary files. As I told, we are not using it for video, but it might still become an issue with large textures, and I guess blend files can grow large too with complex models and animation data. I have been following (lurking over the shoulders :) the commit processes, and it has seemed ok so far, but surely compression (within our 1GB network in this room) is pointless. Is there an option to turn it off?
Might check out Perforce, thanks for the hint. Also looked a bit into Alienbrain. But I guess more interesting for us is to get open source tools, like Subversion or something else, to work in a suitable manner..
Another multiplatform client that fill good also in gnome is svn-workbench: http://pysvn.tigris.org/
Jar, I had problems with svn-workbench… I’m not at work right now, but I got some errors running it, and ended up switching to the command line instead. Still, it’s an attractive gui, and I think some people might appreciate it. (have to solve the issues first)
It all seems to be coming along nicely; will there be an Orange presence at the Blender Conference?
I’m very much a Blender newbie but I do appreciate the significance of what Orange represents and I’m thinking of making the trip across from London for the Conference – will there be sneak previews of the movie?
Toni: do whatever you want, just don’t use Alienbrain. I’ve yet to see any person who uses it and likes it at the same time :)
For a related discussion, see http://lists.midnightryder.com/pipermail/sweng-gamedev-midnightryder.com/2005-March/003038.html thead (SCM to manage binary asset).
kemikazi: Yep, there sure will be! We will have a couple of sessions presenting work, tips, info whatever. Check the schedule for more details: http://www.blender.org/cms/Program.666.0.html
The conferences are usually a lot of fun anyway, very worthwhile to come!
I seem to remember talk of an XML file format for blender…if that ever happens I imagine that being able to diff would be relatively easy…
I think it would be good to look at integrating general support for a few different source control programs (VSS, CSS, Subversion, etc.)…as far as inteface goes, something along the lines of File->Open from Server, File->Save to Server…with some nice interface to quickly view any differences from the new/old files…
Also perhaps moving to (or allowing the use of) project files (I’m thinking more along the lines of each scene having it’s own .blendscene file, or .blendmesh files with a main .blendproject file) it would be easier to check out/in large projects…especially when working over the net)…
George
Can you comment on how Subversion worked out for you during the project? I am curious as to your findings.