Software Engineering Practicum, Semester-in-Review

This will be the last post included in the pdf archive of this blog.  Thus, a brief review of the semester follows.

I thought the Teaching Open Source book was quite useful.  Although we'd already covered some, if not all, of the content it presents in other classes, it still provided useful information and good practice.

Our first-choice project, Gnome Empathy, turned out to be too difficult to actually work with.  We learned some valuable lessons from it: don't assume that an old, well-established project will be easy to contribute to; be willing to change your initial assumptions; don't hesitate to reach out to the community-at-large for some help.

Our second-choice project, OpenKinect, was much newer (~4 months old) when we switched to it.  This meant that there were no simple, quick-fix bugs available for us to work our way into the project.  We dove into the deep end and started learning what we needed in order to be able to contribute something by the end of the semester.  We learned enough to have contributed both code and documentation (both in code and higher-level on the wiki). 

The project as a whole would benefit greatly from a 362/462 group devoted to developing a fundamental structure and test suite.  We've developed a more useful separation of concerns model that we can provide to the project, but perhaps not before the end of the semester.

The class as a whole was definitely worthwhile.  It certainly emphasizes the need for implementing software engineering practices in all aspects of software development.  Without some kind of plan in advance, projects quickly devolve into disparate code pieces smashed together inelegantly.  The result might work, but it also might not.  A plan may change, but the end result is always predicted and expected.

The Next Two Days

It turns out that the OpenKinect.org wiki is a wiki in the truest sense.  One need only create an account and edit and/or create a page to add to it.  So, I took the liberty of updating the high-level API documentation with the data James compiled from the current code.

The structure of the code seems to have changed quite a bit from its initial state, and if we make the changes we have slated on our group wiki, it will have changed again almost back to where it started.  I think it should be pretty easy to accomplish those changes, but we're still waiting to hear back from the OpenKinect community about whether they're welcome changes or not.

We'll definitely have a pull-request for the code and documentation we've added by Thursday (probably tomorrow before class) as well as the wiki in .pdf form.  With only two days of class left, we'll be focusing on the Prezi during class time.

The end is Nigh

With a very short time remaining in the semester, our group is nearing completion on our intended contributions to the OpenKinect project.  I've got a list of planned changes to the API and assorted c files.  The separation of concerns in cameras.c is laughable; James' proposed changes to cameras.c includes creating several new files that are cohesive units instead of the mess that is cameras.c.  I'll be making a a wiki page to document the current state of cameras.c and our proposed changes.  After that's stable, I'll send it to the OpenKinect wiki for addition to the fairly underrepresented documentation section.

Another Contribution

During our coding and documenting adventures we came to the conclusion that the code is quite poorly organized.  So, another contribution to the OpenKinect project that we intend to put forth is a code rearrangement.  The original code was intended to be a quick hack to provide proof of concept.  We've progressed farther than proof of concept, so we need to make it much more clear.  Separating functions into separate c files is the best way to enhance clarity.  James has made a great blueprint for the separation of concepts which I think the OpenKinect community should embrace.

Today, I think we'll work on our presentation and some documentation.  Tim is sick, so he won't be here today, and yesterday was Ashley's birthday; considering he did not turn 21, he shouldn't be too hungover to be at class, but I haven't seen him yet today.

Down, but not Out

So this past Saturday I came down with something that sapped all my energy.  I was in bed for 3 days, and rather than being really relaxing, I was exhausted the whole time.  Finally, Tuesday afternoon I was alive enough to move around, and all was well with the world again.  Needless to say, I didn't get anything done during those few days. 

I noticed that my group-mates came up with some additions to the project presentation page on the group wiki.  Soon, I'll start a Prezi to accommodate all our brilliant ideas.  Prezis are very cool; for anybody who hasn't seen one before, you should check it out.

A Meeting of the Minds

After a very productive group meeting this past Tuesday, our group came up with some useful additions to our wiki page.  We added a documentation goal to be achieved by the end of the third week of April.  We will add the comments from the header file (they basically describe what the functions do) to the c files to which they apply.  Also, we'd like to contribute to the project wiki's documentation page, which currently is pretty unimpressive.

We also came up with a very rough outline for our presentation.  We're (possibly) going to start with a discussion of how not to contribute to an open source project (aka lessons learned) from our experiences with the Gnome Empathy chat program.  Then we'll move on to a discussion of the OpenKinect project, discussing our contributions (code and documentation) and providing a demonstration of what we've accomplished.

The Best Laid Plans of Mice...

I actually managed to see all but one of the presentations I intended to see.  I didn't talk to all the people I had originally intended to, but, in hindsight, I suppose I could have.  Instead of attending Nathan Marz' presentation on efficiency, I went to Steve Sokol's presentation on Asterisk, an open source communications software.  The hindsight comes from realizing I could have still found Mr. Marz sometime other than after his presentation.

Like most of the attendees, I talked with the 3D printing guys.  That technology is pretty amazing and relatively inexpensive.  I particularly like that it can print its own component parts.

My favorite presentation was Walter Bender's learning to learn.  He was very passionate about breaking the classroom mold and doing what helps students learn.  Learning is an activity that students do, not something that is done to them.

John Diamond confirmed my suspicion that open source gaming isn't of the same quality as commercial gaming.  However, Alien Arena, in practice, is really quite good.  Some of the improvements he's made to the Quake II engine are quite impressive.

Bryan Johns presentation on open communications was informative as well.  When I read the abstract for his talk, I was picturing a personal VoIP service, like Vonage.  Asterisk and Digium think much bigger than that.  They provide VoIP services, and more, on a company-wide scale.

On the whole, POSSCON was quite interesting.  If it had been in Charleston and/or I could have attended all three days, I think I would have enjoyed it even more.

POSSCON Plans, Part: the Second

I've still not thought of good questions to ask the people who interest me.  I'm going to attend their talks and seek inspiration there.  I'll be posting the talks I plan to attend now.  The speakers I want to "interview" are in bold typeface.

Chris Hinkley will be presenting Web Hosting – Knocking Out Application Layer & Open Source Threat from 10:00 to 10:45.

John Diamond will be presenting about Open Source Gaming from 11:00 to 11:45.  

Walter Bender will be presenting Learning to Learn: Using & Modifying the Sugar Platform from 1:15 to 2:00.

Bryan Johns will be presenting The Case for Open Communications from 2:15 to 3:00.

David Duggins will be presenting Starting and Running a Business on-the-cheap with Open Source from 3:15 to 4:00.

Nathan Marz will be presenting Become Efficient or Die: the Story of BackType's Success from 4:10 to 4:50.




And, of course, I'll be attending the Tablet giveaway, just in case.

POSSCON Plans

In order to get to Columbia by 9:00, we've got to leave here (Charleston) uncomfortably early.  Whining about getting up early isn't the overall point of this post, however.

John Diamond will be speaking about open source gaming.  He is the CEO and lead developer of COR Entertainment LLC, a company that was initially formed in 2006 that developed the popular open source game Alien Arena.

Bryan Johns will be speaking about open source solutions in telecommunications.  Bryan spent nearly 20 years in and around the businesses of technology and telecommunications.  He has started, grown and sold a handful of web application development and VoIP technology businesses and in 2004, found a home in the disruptive world of open standards and open source telecommunications platforms.

Nathan Marz will be speaking about big data.  Nathan Marz is the lead engineer at BackType where he builds analytics tools for social media.  BackType collects many terabytes of data from Twitter, Facebook, social news sites, millions of blogs, and provides analytics on this data in real-time.

All three of these topics interest me to some degree, big data and gaming more so than telecommunications.  Over the next few days I will ponder some questions to ask them.

OpenKinect: Documentation and POSSCON

The source code for our chosen project is documented fairly well already.  I think we should add some comments to the code in the few places that lack an explanation of the function/segment of code.  Further, I think we should document our additions to the code with more in-depth comments for those with little experience with C.  A working source with good documentation can effective for learning a new language.

POSSCON is coming up soon.  I'm looking forward to that seeing how professionals use and contribute to the open source community.  In a later post I'll cover the people I intend to talk to at the convention and the questions I want to ask them.

Catchup

It seems I was remiss in my blogging.  I completely forgot to blog about TOS chapter 8.  Chapter 8 is titled Explaining the Code, but it's really about documentation; I suppose the two aren't mutually exclusive.  The text describes two types of documentation: ad hoc and planned. 

Basically ad hoc documentation occurs during the working process.  Any code that gets developed must have documentation that explains the reasoning and any unclear code.  We've learned good practices to use to develop sufficient ad hoc documentation in every programming class we've taken.

Planned documentation is the development of a formal document that fully explains a piece of software.  This type of document is of much higher quality than the ad hoc documentation.  We learned how to generate these kinds of documents in Software Architecture and Design.

An important concept the chapter addresses is the waterfall method approach to technical writing.  It outlines

 

1. Planning -- who is the audience? What are the book's goals?

2. Content -- what are the chapters about? Where will you get the information?

3. Writing -- first draft, review, second draft ...

4. Internationalization/Localization -- will the book be translated? Into what languages?

5. Review -- what worked? What didn't? How will the book be maintained? 

 

and says that starting with the first and proceeding down is the best approach.

OpenKinect Schedule

After reading through James' most recent blog post, I've created and posted a rough schedule for the rest of the semester on the project wiki.  It isn't very detailed yet, but it does provide a good outline for what we've done and what we will be doing.  As we make progress, we can fill in the details, which registers we'll focus on at any given time, for example.

After some initial discomfort with C programming and low-level hardware driver manipulation, James' amazing progress and coding skill have swayed me to the feasibility of actually succeeding with this project.  We've already made some non-trivial contributions, and the more we accomplish, the better off the project will be as a whole.  I'm looking forward to seeing what we can do by the end of the semester.

The cost of boredom

Tuesday I decided to do the exercises for TOS Chapter 7 because I was at a loss for something else to do.  Now today I'm finding myself without anything about which to blog.  I updated our team's wiki page to reflect our decisions regarding our "bug" and semester contribution to the OpenKinect project.  Briefly, we need to figure out the USB Control Command structure, write a USB control command for the OpenKinect libfreenect library, and then use it to manipulate the RGB camera in the Kinect controller.  Hopefully, it'll be as easily done as said, but I'm doubtful it'll be simple.

TOS Chapter 7

Chapter 7 of the Teaching Open Source book is about patches.  In short, patches involve running the diff command on a file that has been changed.

Exercise 7.2.2
The -u option to the diff command makes for a much more readable output, and it even includes some context (which isn't strictly necessary since "hunks" of code are labeled with line numbers.)  The diff command without the -u option is still useful, it's just not as easy to read.

In the initial, simplistic example, only one file was modified and patched.  A more comprehensive example followed which made a change to two files in a directory and a diff was applied recursively to the directories.  This captured the changes to all files within the directory, which will be much more useful when working with huge source code directories.

Both the individual file and the directory diff patch methods are not using an SCM.  Using an SCM to create a patch is almost identical, except, instead of using two different files/directories, the SCM compares changes in the current working copy against the most recent revision.  The latest HEAD code should always be used when making a patch, so that the development team can more easily integrate the patched changes.

Applying a patch is a simple matter.  It's similar to the shell redirect described in the initial diff section, only instead of redirecting the output of a command to a file, a file is redirected to the input of a command.  That command is, shockingly, patch.  Running patch from the directory in which to make changes seems easiest.  The option exists to run the command from root, but I don't fully understand the -p flags.  From the text it sounded like I could strip off x number of directories from the paths listed in the patch, but if I'm running the patch from the root directory, wouldn't those files then exist in the root directory?  I'll have to remember to ask about this in class.

I think I missed the point of Exercise 7.8.  Creating an empty file called foo, and then running diff on that new file against /dev/null resulted in no output from the diff.  I imagine that's because both files are empty and there's no difference between them.
So I asked about the exercise in class; turns out, the fonts on my screen look too similar to tell apart.  I was supposed to create an empty file called foo and put "bar" as the contents of the file.  This would have yielded some output from diff -u when run against /dev/null.

Exercise 7.9 was almost identical to the example for running diff on a single file from the very beginning of the chapter.  The only difference was the file was located at the end of a path, instead of in the working directory.  The patch file was formatted exactly the same.

Exercise 7.10 tells us to make a patch for our selected project.  We're steadily working on that, and I'm convinced we'll have this feature request completed "soon."

5th Annual CofC CS Alumni Symposium

Tuesday, February 15th, 2011.  A series of computer science alumni present various topics ranging from getting hired after graduation to advice on how to be a successful graduate student.  I enjoyed seeing people I actually had classes with presenting at the alumni symposium.  Clearly, the CofC CS department prepares us for success.

Some tidbits of useful tips I gleaned from the presentation: have an interest outside the computer science field, play a musical instrument, have a wide breadth of knowledge in a small company and a large depth of knowledge in a small company, have a specific interest within the computer science field, get involved in a project in that interest, have excellent communication skills, read about CS current events, and my favorite, get enough sleep.

The bug you know

Ashley found a bug that he feels we will be able to fix.  "Bug" in this sense is actually a feature request, but we feel that qualifies.  The gist of the bug is that the RGB camera has autoexposure, and at least one user wants to have manual control of gain and exposure.  There's documentation related to this problem.

I noticed in Ashley's blog a comment from Drew Fisher, one of the people who reverse engineered the camera protocol.  He's currently working on unlocking the camera in the Kinect, including features that the Xbox doesn't currently use.  He's offered to help us with questions involving libfreenect, and the Kinect controller.

Building OpenKinect and TOSS Ch6

Building OpenKinect
Thankfully, our new project is being developed on the current release of Ubuntu.  The instructions for downloading the OpenKinect source code and building it are quite accurate, as of today.  There is a note that says the instructions are subject to change, but with the project only being a few months old, I guess there haven't been many drastic dependency changes yet.  All told, this project took about 3 minutes to download and build. 

TOSS Ch 6
4. One problem I have with our new project is that the only bug-tracker I've found is Github issues, a sub-page of the project's github site.  There are only 2 bugs listed there, and they're both 2 months old.  That's two-thirds of the project's life to date.  The oldest is related to packet loss in the depth view.

5. Creating a github account was a simple task.  Click login, select pricing options, select free for open-source, input the data, and you're done.

6. I don't personally have a Kinect controller, so I will have to rely on Ashley to reproduce the bugs we find.

7. Another severe limitation with github issues is that it's not really a bug-tracker.  According to FOSS chapter 6, a bug-tracker should have a Summary, a Description, Comments, a Reporter, an Owner, a Version, a Severity and Priority, a Status, and a Resolution.  Github allows Reporters to tag their issues as bugs and they can write a description of the bug.  Other users are able to leave comments on those bugs.  Unless the Reporter has foresight, the Version could be completely left out, and Owner, Severity/Priority, Status, and Resolution aren't even addressed at all.  With these limitations, performing triage on our project's bug list would require porting all the information to a real bug tracker, like Bugzilla for instance.

Despite the failings of Github as a bug-tracker, we can at least build the source, and with Ashley's Kinect controller, we can attempt to replicate and test the reported bugs.

Building Freeciv

I'm happy and sad all at once.  Building Freeciv was so easy as to be almost mindless.  Certainly, there were some bugs to be found, but compared to trying to build Empathy from the source available, Freeciv was simple.

There were some dependencies missing from my system, but the Package Manager was more than able to take care of finding them.  I tried using apt-get, but they had some odd names compared to the clues the autogen.sh script gave.  Also, I think many of the packages I had to install for Empathy were ones required by Freeciv, so I had a lot less to search for during this build process.

Problems aside, this build process was how I was hoping Empathy would have gone.  Perhaps it makes sense that the homework assignment, meant for only a couple days of effort, would take so much less time/effort to compile and run than the semester-long project did.

As a small update to the Empathy problem, it's still a problem.  As yet, I've been unable to get Ubuntu 11.04 to install, even on a virtual machine.  I've not yet given up hope, but success is looking less and less likely.

Building Empathy

Downloading Empathy was a breeze.  Using Github was no different from using the command line with Subversion.

Building Empathy from that downloaded source has proven to be most infuriating.  The source depends on libraries that aren't released yet, so in order to develop the code, one must first find the as yet unreleased libraries.  The up-side to finding these libraries is they're part of the alpha release of Ubuntu 11.04.  The downside is, to get them, you have to use an alpha release of an operating system.s

It's possible to get Empathy from the Ubuntu Software Center and run it from there; however, this won't help us see our changes once we start adjusting the code.  Unless one of my teammates has found success in the build/install process, we may have to change our project.

Subversion

Subversion turns out to be a very powerful, yet simple tool.  The Windows tool TortoiseSVN adds commands to windows explorer menus when you right-click on files or folders.  These commands do everything from checking out a repository to committing changes made.  It also provides an icon to indicate the status of files while in windows explorer. 

The Ubuntu tool  RapidSVN is a GUI that has all the same features Tortoise provides, but it's done through a GUI instead of explorer menus.  Ubuntu also provides command line tools for maintaining an SVN repository.

Setting up a Subversion server was also quite simple.  I used VisualSVN server to set up a simple SVN server on my Windows machine.  After installing, it was simple to add a new repository, add a user, and connect to the repository using those new user credentials.

The History of GNOME Empathy

This weekend I researched the history of the GNOME Empathy project.  It started as a reinterpretation of a project that was a compilation of two other projects.  Needless to say, the history wasn't exactly clear.  It is a messaging program integrated into the GNOME desktop.  It's written in C, which will be interesting because most of our group aren't familiar with C programming.  The project is sponsored by Collabora Ltd., a software consultancy based in Cambridge, England.  The project leaders are Guillaume Desmottes and Xavier Claessens.

The Cathedral and the Bazaar and IRC

I'm going to start this by extracting all the rules of open-source development from the essay and making a list.

1. Every good work of software starts by scratching a developer's personal itch.
2. Good programmers know what to write. Great ones know what to rewrite (and reuse).
3. "Plan to throw one away; you will, anyhow." (Fred Brooks, The Mythical Man-Month, Chapter 11)
4. If you have the right attitude, interesting problems will find you.
5. When you lose interest in a program, your last duty to it is to hand it off to a competent successor.
6. Treating your users as co-developers is your least-hassle route to rapid code improvement and effective debugging.
7. Release early. Release often. And listen to your customers.
8. Given a large enough beta-tester and co-developer base, almost every problem will be characterized quickly and the fix obvious to someone.
9.  Smart data structures and dumb code works a lot better than the other way around.
10.  If you treat your beta-testers as if they're your most valuable resource, they will respond by becoming your most valuable resource.
11. The next best thing to having good ideas is recognizing good ideas from your users. Sometimes the latter is better.
12. Often, the most striking and innovative solutions come from realizing that your concept of the problem was wrong.
13. "Perfection (in design) is achieved not when there is nothing more to add, but rather when there is nothing more to take away."
14. Any tool should be useful in the expected way, but a truly great tool lends itself to uses you never expected.
15. When writing gateway software of any kind, take pains to disturb the data stream as little as possible—and never throw away information unless the recipient forces you to!
16. When your language is nowhere near Turing-complete, syntactic sugar can be your friend.
17. A security system is only as secure as its secret. Beware of pseudo-secrets.
18. To solve an interesting problem, start by finding a problem that is interesting to you.
19. Provided the development coordinator has a communications medium at least as good as the Internet, and knows how to lead without coercion, many heads are inevitably better than one.

This list of rules outlines the sound strategies Raymond implemented during a successful attempt at starting an open-source project.  He then documented his thoughts and motivations during the execution of that project.  He modeled his attempt after what he perceived as Linus Torvalds' successes with the Linux OS.

Another list worthy of note ... the closed-source program manager's five functions:

1. To define goals and keep everybody pointed in the same direction
2. To monitor and make sure crucial details don't get skipped
3. To motivate people to do boring but necessary drudgework
4. To organize the deployment of people for best productivity
5. To marshal resources needed to sustain the project

Raymond tries to explain why these functions are necessary, but instead explains why open-source makes them unnecessary.  They don't need to marshal (defend) resources because everybody brings his own toys.  They don't need to organize for best productivity because only the best of the best get any attention in the elitist open-source culture.  There's no need to motivate or monitor people who are volunteering to do the work in the first place.  The goals of open-source projects are elicited by the leaders of the project and elaborated upon by the beta-users and co-developers, rather than a committee that gets the goals wrong 60 - 75% of the time anyway.

This essay has pretty much sold me on the effectiveness of open-source software development.  Toward the end of the essay he says, "Human beings generally take pleasure in a task when it falls in a sort of optimal-challenge zone; not so easy as to be boring, not too hard to achieve. A happy programmer is one who is neither underutilized nor weighed down with ill-formulated goals and stressful process friction. Enjoyment predicts efficiency."


IRC was remarkably easy to set up.  I'm currently using the windows version of Gnome Empathy, which supports IRC as a chat method.  A quick search of the Pidgin website detailed how to set up an IRC room.  Joining the mailing list was also quite easy.

Group Project Picks

Last Thursday, as a group, we chose 3 FOSS projects that sounded interesting.  After doing some research on them, I've decided the Gnome Empathy project has the best support for new contributors.  They give suggestions for developers, provide links to the source and details on how to build it, and have both IRC and a mailing list for contacting the team.

The other 2 projects the team chose are both interesting. 
The OpenRemote project looks to be nearly complete, and it doesn't have much in the way of documentation on their website.  Of course, it could have all the necessary documentation bundled with the source code.  They also don't have any obvious IRC channel, but they do provide a web-based chat-room with logs of previous chats.

The OpenKinect project also sounds interesting, but it's not software in and of itself.  It's software to allow for integration of the new Kinect controller for the XBox 360 with various operating systems.  Their wiki has a lot of information available and IRC and mailing lists as well.  It also links to the source code and provides detailed build instructions based on operating system.

I hope I'll be able to convince the team to go with Gnome Empathy, but they sounded very excited about the OpenKinect project.

Blogs, Wikis, and POSSCON; Oh my!

Setting up a blog was easy.  Google already knows who I am, so it was just a matter of asking them to set up a blog for me to use.  Setting up a Wiki for our group, called When It's Done, was easy as well.  The wikispaces website is easy to use, and simple to operate.  Signing up for POSSCON was also easy (confirmation info: Transaction ID: RKDRJGQZCF, Order Number: 10000239), and free to boot.  I have Physics on MWF, so I don't plan to attend Friday's events for POSSCON, but I will be there for Thursday.

I read chapters 1 and 2 of TOS.  The information inside was mostly stuff we've either gone over in class, or I've seen it somewhere before.  I imagine it'll become much more useful quite soon.