This last week saw great advances for the CanDo package of SchoolTool.  It started with a meeting in Richmond, Virginia, with the state administration regarding the up-coming deployment of CanDo in eight new pilot sites across the state.  All parties agreed that we were to go ahead with the deployment this Fall.  It was decided that support would be handled by the state office with coordination back with the Arlington Career Center when needed.  The priority for reliability and ease of use will greatly drive the upcoming summer development effort.  Lastly, we had the opportunity to discuss some of the technical issues regarding competencies which, following the meeting,  led us to make some very important design decisions.

After the meeting with the state, we gathered back at the Arlington Career Center to plan the development sprint that Alan Elkner and Eldar Omuraliev undertook this last week.  Together with David Welsh and Jason Straw, Alan and Eldar discussed a new design for course and section competencies.  The first issue addressed was removing the need for competency inheritance, a source of customer confusion and occasional data-integrity problems.  The second issue was how best to address year to year changes in the global competencies that the state publishes and the complexity of reimporting those changes into an existing CanDo deployment.

The good news was that one solution will be able to address both issues.  Since the state will always supply us with global ids for the competencies they publish, we can reliably use them as keys for the course and section competencies.  Secondly, the fact that these global ids will never change regardless of how they may be organized into the competency groups, we will not need to worry if the structure of those competency groups changes from year to year.  By keeping a list of global competency ids attached to courses and sections, and not keeping any references to groups, we have the ability to deliver those competencies to the grade book with complete reliability.  Bottom line, no need for inheritance and no difficulties reimporting.

The rest of the week involved the actual coding sprint.  The first task was to integrate CanDo into the new SchoolTool tabbed interface and, in the process, remove the need for specifying the CanDo skin in the url or CanDo requests.  The integration took only a couple of days, and now CanDo looks even cleaner than before.  The seem-less integration with SchoolTool will allow users to switch between CanDo and the rest of SchoolTool merely by clicking on the tabs at the top.

With the remaining two days of the sprint we got started with the overhaul of the course and section competencies.  The use of global ids is now unit-tested, and the presence of inheritance has been completely removed form the user interface.  Eldar's job for the next month will be to create views for assigning global ids to courses and sections and for creating the leaner and cleaner grade book that uses them.  The new grade book will not only take advantage of the more reliable competency design, but it will also reduce the time to load by only loading those competencies of the most recently graded competency group.

In all, it was an exciting and highly productive week, and Eldar has a clear understanding of what he needs to do for the next month.  When we gather for the next development sprint in June, we will be bringing last year's interns aboard as junior developers, and we will be able to focus intensely on writing comprehensive functional tests for the entire CanDo application.  This will allow us to address the high priority issue of application reliability favored by the folks at the State of Virginia as well as allow us to move forward with any new features with complete confidence.  In short, we will be ready for the Fall deployment.

SchoolTool Spring 2008 (Hardy) Release Notes

• Installation and updating via Ubuntu package management.
• Proper init scripts for controlling startup/shutdown.
• Installation conforms to Debian practices; files are in standard locations.
• Custom access control policy for SchoolTool allocates permissions based on group membership or relationships.
• Access control partially customizable through the web.
• Automatic generation of large-scale sample data for testing.
• Data import through CSV files or custom Python scripts (LDAP and CAS single sign on support under development).
• Server control and database maintenance via the web.

• Uses Ubuntu package-managed Python 2.4, Zope 3.4 and Paste.
• Full i18n and unicode support throughout.  Existing translations are out of date, but will be updated in coming months via Rosetta on Launchpad.net.
• SchoolTool is now a WSGI application, allowing the use of any WSGI-compliant web server, including Apache with mod_wsgi, and WSGI middleware.

What was most remarkable about the SchoolTool sprint at PyCon was how routine it was, especially considering we had a team of three adults and four high school students.  The students, Andrew, Chris, Filip, and Willi are all veterans of sprints on the CanDo project and got right down to work the first morning of the sprint.  Filip gets bonus points for taking a cab to a nearby Radio Shack to buy a soldering iron to fix his power cord prior to the beginning of the sprint. Also, Ignas's buildout configuration made it much easier for everyone to hit the ground running with a working set of packages.

The main goal for the sprint was to finally break out of SchoolTool's traditional navigation paradigm based around traversing the ZODB's object hierarchy and establish a more task based system.  For example, if a teacher wants to do their grades, they should be able to just click on a prominent "grades" tab or link and get to work.

The hard-won lesson from previous sprints is that the navigational changes need to be done in a backward compatible way.  If every single view in the system has to be touched and/or redesigned to complete the transition, it is impossible to complete the job at the sprint, and
difficult to maintain the momentum to complete the job in the face of other pressing local needs.  So while the new navigation system is not "done," it is functional, and will be used in our test schools and upcoming releases.  It can and will be prettier in the future, and there are numerous refinements and finishes necessary, but I don't anticipate another full scale reorganization for a long, long time.

A couple observations from after we made the initial changes.  Jeff Elkner looked at the main calendar page, which he's presumably seen dozens of times over the past couple years (at least) and said "Hey, you can print out PDF's and export to iCal."  Of course, you've been able to do those things all along, they're just more noticeable now.  So that looks like an improvement.

Also, Ignas observed that not having breadcrumbs and and  navigation menu hanging around will take away some crutches that let us get away with not thinking about the user experience.  It was too easy to let the user find his or her way out of traps with the breadcrumbs, but every time that happened, it was really a failure of usability.

Ignas also prepared a queue of nagging, mostly calendar-related, bugs to keep everyone busy throughout the three day sprint.  Ignas did an excellent job of organizing and leading development at the sprint and getting the best out of our interns.

Removed breadcrumbs (Ignas).
Moved navigation menu to top (Ignas).
Made actions menu row of buttons (Ignas).
Yearly calendar view displays term information (Filip).
Made SchoolTool more beautiful in many small ways (Andrew).
Creation of a manage tab, centralizing management screens (Ignas).
Fixes to timetabling management UI (William).
Fixed score system bug in add activities view (Paul).
Added a home view for persons (William).
Removed use of deprecated zapi (Alan).
Refactored calendar portlet to be viewlet based (Filip).
Significantly improved weekly calendar view with timetable integration (Chris).
Improved SchoolTool resource booking view (Andrew).

Also, we'd like to thank Harper Reed of skinnyCorp for showing us around Threadless HQ.  It was great for the interns to see the inside of a wacky web startup.

Ignas Mikalajūnas has overhauled the SchoolTool build system to automate the process of fetching and building dependencies.  The updated instructions assume you're on Ubuntu or another .deb based Linux distribution.

Overview

The basic changes in process we made this year were effective. Development has moved forward much more steadily and consistently than in the past and more closely aligned with the immediate needs of schools. Local funding of active CanDo development has boosted our base of developers and testers, addressing our most pressing constraints. Our school/developer relationship in Brussels did not work out, at more of a cost of time than money.

Code

We've now got running code being tested or used in production for our major components:

• Demographics: Ignas wrote a simpler demographics package to meet the needs of the Vilnius Lyceum and CanDo. It is being used at both sites. There is still a lot of functionality to be built on this base--primarily reports--but the foundation is solid.
• Gradebook: Alan Elkner picked up development of the gradebook module Stephan Richter started last year. It is currently used by Jeff Elkner. Ian Benson will be using it to gather data from maths teachers in eight schools in the UK using his Tizard curriculum. The CanDo competency-based gradebook is also being used in production in the Arlington Career Academy.
• Attendance: Ignas has implemented a "journal" attendance and grading application for the Vilnius Lyceum, which allows teachers to enter a grade for each meeting of the class or mark and absence. The Computer Science department at the Lyceum is currently testing this. Going forward, this will become the standard SchoolTool attendance system.
• Calendaring: The calendaring code has been cleaned up and improved with input from CanDo and testing at Vilnius Lyceum. It needs more testing and tightening in real use.

Other ongoing code issues:

• Interoperability. We funded initial LDAP integration via the Tizard project; Ignas is doing further work on it for use at Vilnius Lyceum. We are planning on supporting CAS for single-sign-on with popular PHP apps used in schools like Moodle. We've got a simple CAS implementation already using existing open source code.
• Releases. As you know, the Zope 3 world was in turmoil this year, especially regarding packaging, and particularly for us the (non-)release of Zope 3.4. Once it is released we will stick with Zope 3.4 as long as possible, because a lot of the burden of packaging Zope 3 and related components for Ubuntu and Debian has fallen upon us.  We have worked under Brian Sutherland's direction on a set of Ubuntu packages for the past six months. We had a set of packages ready for Gutsy but they could not be included in the distribution due to the non-release of Zope 3.4. We subsequently created a new snapshot and developed some scripts to generate packages using the Launchpad PPA, which makes the packaging process much easier going forward. Unfortunately, we also discovered an arcane bug in the PPA, which is now being addressed, but in the meantime, we haven't done a full public release. Since our installation story is now just "apt-get install schooltool", there is no sense in making a release until the packages work.

Big pieces missing from the code:

• Dealing with the passage of time, terms, history. For example, when a student transfers between classes during a term, they can't simply be deleted from the old section, we need to retain the records from that section and keep it integrated in reports for the term. These are issues that touch all parts of the application and will be Ignas's main development priority in 2008.
• Reports. A general release of SchoolTool will at minimum have to have a large set of standard report templates. In 2008 we'll be writing a lot of custom reports for partner schools and deriving standard ones for general distribution.
• More ways to get data in and out from other applications.

Developers

Ignas Mikalajūnas: Switching Ignas to working full time on SchoolTool and acting as "lead developer" and maintainer has been immensely helpful in getting the project back on solid footing. Put simply, SchoolTool now functions like an organized open source development project, not a loose aggregation of contractors managed by a teacher. Ignas has been available and responsive to me and other participants in the project and demonstrated strong leadership when needed.

Jean-Francois Roche: When we started working with Jean-Francois, we were at a point where we needed to find both a developer and a local school. I think we did ok with Jean-Francois, but the school he (and Nicolas Pettiaux) found didn't pan out. They just weren't sufficiently engaged in the project, and Jean-Francois ended up with an unmotivated and uncooperative "customer." This caused repeated delays, required increasing nagging to get information from the school, and with plenty of other (higher-paying) work he could do, Jean-Francois just ended up spending less time on SchoolTool than we would have liked him to. The work he did do was fine, and his billing was in line with the work completed.

After our meeting at the beginning of November, Jean-Francois and I decided to dissolve the partnership with the school, I gave him some work to do on the SchoolTool calendar for the rest of the year, and we left open the possibility of part-time work on SchoolTool next year.

Alan Elkner: An experienced Java developer, Alan started working on learning Zope 3 and SchoolTool, and open source development processes, about a year ago. During the summer he did substantial paid work on CanDo to close up bugs and add features before and after the beginning of school. After CanDo's budget was exhausted, I started paying him with unused development funds to continue CanDo bugfixes (some of which are SchoolTool bugs regardless) as necessary and work on the SchoolTool gradebook.

Once he climbed the Zope 3 learning curve, Alan became a very useful and productive member of the team. He has the right disposition for following through with this work. He's older, not interested in developing new web framework features, just used to writing functional, unsexy software for users. He takes Ignas's advice and direction well. His steady reliability is exactly what we need now.

CanDo Interns: We funded $10,000 worth of work on CanDo by student interns. Jeff Elkner ran a very active program through the spring to train high school students to do SchoolTool and CanDo development. About three students are genuinely productive Zope 3 developers (which, as you know, is hardly easy). Given the scarcity of Zope 3 developers and the effort already spent training them, we are trying to keep them engaged.

Schools

Vilnius Lyceum

Ignas has primarily worked with the three teachers in the computer science department at the school, which is probably sufficient right now. When I met with them in Vilnius, it was clear that having local users had given Ignas a much more focused sense of the requirements and how reality conflicted with his "beautiful abstractions." I pushed to have Ignas visit the school weekly to get feedback in person, and that seems to be helpful as well. Given that Ignas will also be responsible for the remaining chunk of core development and organizing releases in 2008, keeping a small but active group of testers at Lyceum is more important than scaling up to the whole school.

Lycée Emile Jacqmain

This is the school in Brussels. We have stopped working with them due to a lack of interest on their part.

Science Leadership Academy

Science Leadership Academy (SLA) in Philadelphia was one of the schools we hoped to partner with in 2006, when we found that partnership didn't work without a local developer. It is essentially the perfect school for us to work with in terms of capacity. It is a small, innovative school with a 1-to-1 laptop program, a sys admin on site and good professional development support. Most importantly, the principal, Chris Lehmann, developed open source administrative software (in PHP) as a teacher at his former school. As is usually the case, it was too specific to that school to be portable to his new school, but he not only knows exactly what he needs, he can often show you an example.

The good news is that Alan Elkner moved to Philadelphia this fall, so we now have our local developer. I flew down last month (another advantage, I can easily fly there and back in a day) and Alan, Chris and I met. They are both excited about working together. I know Chris will push hard to get SchoolTool to do what he needs and deploy it school-wide in the fall. In particular, he is interested in student and parent access to data on SchoolTool and basic integration, including single sign on, with Moodle and Drupal.

Tizard Schools

Ian Benson approached us via The Shuttleworth Foundation, interested in using SchoolTool to help gather data from research he's conducting using his maths curriculum. In the spring we funded his team's work to add LDAP integration to SchoolTool, allowing him to sync data back to his LDAP server from the schools. We are currently funding their packaging of SchoolTool for distribution and deployment to 8 test schools in the UK. I don't like operating at an additional level of remove from the actual schools, but since they have some development capacity, and I had funds available, we've tried this route.

The deployment is currently behind schedule. Hopefully it will be completed before the end of the calendar year so payments won't be rolled over to the next year.

2008 Roadmap

Bug tracking, feature requests, translations and FAQ for SchoolTool are now managed on Launchpad.  The SchoolTool page on Launchpad is here.  The CanDo page is here.

To test the new SchoolTool Ubuntu packages for Feisty, do the following (if you're using Ubuntu Feisty, natch):

1. add the following line to the end of your /etc/apt/sources.list
   deb http://ftp.schooltool.org/schooltool/archives/ubuntu feisty main
2. sudo apt-get update
3. sudo apt-get install schooltool
4. say "y" when it asks to install unauthenticated packages
5. go to http://localhost:7080 (user:manager, passwd:schooltool)

That's it. Debian packaging is a pain to set up, but it is a beautiful thing when it just works.

There were three goals for the sprint:

1. Packaging a SchoolTool release for Ubuntu Gutsy.
2. Making some improvements to the fit and finish of the application.
3. Paying down some technical debt around terms and timetable implementation.

We achieved all of our goals.  It is difficult to appreciate the complexity of packaging an sever application like SchoolTool so that it can be installed "apt-get install schooltool."  SchoolTool is built on the Zope 3 component architecture, and re-uses a number of associated components.  Unfortunately, while re-using components saves us some development time, it falls upon us to create packages for each of them for Ubuntu and Debian (and we're not even talking about other distros and OS's right now).  Our intrepid volunteer release manager, Brian Sutherland, led this considerable effort, with help from Ignas and just about everyone else from time to time.  The initial packages for the release are on our server, and we'll explain how to test them, and the broader goals for the Gutsy release, in a subsequent message.

Andrew Rodriguez is a very talented CanDo intern who happened to be spending the summer in Spain, so he was flown over by CanDo to participate in the sprint, and handled himself impeccably all week, despite being fifteen.  He worked with me on a number of bugs and annoying issues that I wanted to eliminate before the next release.  One thing we had some fun with was "context sensitive Zonki," which means that our Zebra mascot/logo changes depending on what actions the user is undertaking.  Paul, Jean-Francois and Ignas also contributed to this effort. 

A more detailed list, based on the task cards we used at the sprint, can be found below. 

Release

• Functional tests for new BasicPerson objects. -ignas
• SchoolTool package configuration. -jinty
• ZCML files refactoring. -jinty
• Zope 3.4 package. -jinty
• Removing immature components for release. -ignas

Finish

• Person's timetable view disabled by default. -pcardune
• Arrows for drop-down menus. -andrew
• Streamlined section creation. -pcardune
• Updated Zonki favicon & logo. -andrew
• Context sensitive Zonki. -andrew
• Saturday and Sunday are non-school days by default. -jfroche
• robots.txt disallows spidering by default. -andrew & ignas
• Drop down menus require click. -pcardune
• Fix misaligned fields in password edit form. -andrew
• Reorganize edit relationships form for better usability. -andrew
• Fix namechooser to handle blank or otherwise useless titles. -andrew & ignas
• Fixed navigation from the add section view. -andrew

Debt

• Test removal of terms/schemas. -pcardune
• Fixing url quote bug in section search. -ignas
• Fix editing and deletion of terms. -jfroche
• Generations bugs from Cando. -pcardune
• Improve schedule creation forms. -pcardune

I asked Ignas to prepare an informal summary of his work on SchoolTool in the first three months of 2007.  He has been focusing on refinements to existing SchoolTool functionality needed to deploy SchoolTool at the Vilnius Lyceum.  His report (with screenshots!) is here.