FeedsIt’s out, it’s shiny and it’s from a new company.
And the XML handling has been rewritten to comply with my previous gripes. However, at the same time, the schema appears to have changed substantially, so I’ll have to investigate what’s changed and make the necessary changes to the Companion.
Fun times…
I released version 0.4 beta 3 of the DVD Profiler Companion last night – the main changes over the last version being support for the new IMDb page design, and support for Windows Vista.
Users of DVDFolks, are strongly recommended to switch to the Companion, although I have detailed a temporary workaround for the IMDb changes on the DVDFolks page.
In the last 20 minutes I’ve had 4 people email me to say that DVDFolks doesn’t work any more because IMDb have had a redesign…
So if you came looking for a way to contact me about it – please don’t bother.
But thanks for the thought.
I released DVDFolks 0.8 back in May 2005. At that time I said that I wouldn’t be doing any further development work on it, but I would be providing similar functionality as part of the hadn’t-yet-been-started DVD Profiler Companion.
Since then IMDb have changed things around a bit on their pages breaking DVDFolks in various ways. Where possible I have made the necessary compatibility fixes, but episodic TV has been broken for a long time.
Yesterday, I decided that it was actually time to do something about. And this is what I have…
Even in it’s early state, it’s a lot better than DVDFolks ever was – the code doesn’t make me cry when I look at it, it looks a lot prettier, it’s faster and I fixed a couple of bugs I noticed in DVDFolks.
TODO:
Remove hard-coded strings, so that people can translate it…
Add a mechanism for selecting which episodes to extract information for in TV mode (it currently extracts the entire lot, which isn’t particularly useful)
Release a beta
Add a mechanism for editing the extracted names/roles (not a priority for beta 1, since other tools can do this)
Release another beta.
To the 1000+ people who are using DVDFolks – I’m sorry (really!) for the wait.
Personal
1) Release DVD Profiler Companion 0.35
Work
1) Finish tweaking design of new-look e-shop.
2) Add checking of membership information to ‘members-discount-checkout’ for said e-shop.
3) Work out how exactly how I’m going to implement a ‘Student Learning Zone’ thats been in low-level planning (’we want’) for 18 months by mid-November (project budget = £0).
4) Start implementing what I planned in (3).
I made the decision to wait sometime around April 2005. I started coding in late December.
Actually it was December 25. Xmas Day. And it was probably the most productive day I had all year (when it came to doing personal stuff). By the time I went to bed that night I had a working prototype that generated output that was actually useful.
It was so CPU-intensive that it was causing my PC to automatically switch off to stop the CPU suffering heat-damage (no, really – I had to give it an hour to cool down between test runs!), and it crashed a lot. But it worked.
I worked on it intermittently over the next few days, and made it more efficient and less crash-prone. But it still crashed, still used massive amounts of CPU power and still used massive amounts of RAM.
But in January, I released it anyway as version 0.1. The first few people to try it had problems, but minor fixes in 0.1.1 and 0.1.2 improved things considerably. Of course a 0.1 is hardly a finished product, and lots more changes were to come…
So – having made the decision to develop a tool, I needed to decide what platform I wanted to use. Having developed DVDFolks as an HyperText Application I decided that I didn’t want to go down that route again – but I did like writing an application using HTML and JavaScript. I like beign able to skip the ‘compile’ step before running what I create.
My favourite browser (Firefox) is primarily written using XML and JavaScript. So why not do the same thing?
Having made that decision, I was left with 2 options – develop the tool immediately as a Firefox extension (but that would require that all users of the tool used Firefox), or wait for Mozilla to develop XULRunner. XULRunner for Mozilla, is what .NET is for Microsoft – a ‘library’ of functionality that applications use as a foundation. For my as-yet-unnamed-tool, that primarily meant an XML processor, and file reading/writing functions. Although long-planned, XULRunner was (and still is to an extent) not a very high priority amongst core Mozilla coders.
So I waited.
For a very long time, I’d been unhappy with the list of my DVDs here on dougweb.org. For a long time, I merely linked to the IVS MyProfiler page. Updating that was easy, and the list was kept pretty much up to date.
After a while, I used Roolku’s external skin to get a fancier look, but because that requires an XML export, I didn’t update it as frequently. It also meant downloading the entire XML file to view even a single DVD.
I then added a PDF, that was produced by ‘printing’ a DVD Profiler report. That created a file that was even larger than the XML. And because producing/uploading a PDF and the XML took a long time, updates became rare. I updated the list less and less frequently, leading to multiple-month gaps.
And finally, I toyed with using the XML file and and XSL transform to generate custom web pages. That was a less than half-hearted effort.
So I decided that I really needed to create a solution that I was going to be happy with. Enter the DVD Profiler Companion.
Finally!
http://www.dougweb.org/dvdpro/companion/
All sorts of goodies – cross-linking, translations, decent navigation, speed improvements, compatibility enhancements and more – see the changelog for more details.
Over the next few days, I’m going to post some thoughts on the development experience.
Since 0.2.2, I’ve made the following changes. Posting this now, so I have a record of what’s changed before I forget…
Work still to do
Translations still wanted!
Thanks to Christian Rygaard for doing that (the translation will be included in 0.3)
This happened again on Thursday – except this time I didn’t get my hard drive back free of data loss – I seem to have lost my local copy of the DVD Profiler Resource Centre (the entire thing!). What’s curious is that I haven’t found anything else missing yet…
The worst part about the loss, is that my development version of the DVD Profiler Companion lived in that folder, and has therefore dissappeared. At least (most of) the changes were done in the last few weeks, so I’ve got a fairly good idea of what I did between 0.2.2 and Wednesday evening.
To whichever git(s) posted on tech troubleshooting boards that the error I encounted was not symptomatic of a failing hard drive – you were obviously talking shit.
Bah.
One week later than planned, but that’s life.
Now waiting for the bug reports.
Tracked down a leak bug that was stopping further development to another bug in XULRunner (well Spidermonkey). Found out that it’s already fixed, but only on the ‘unstable’ trunk.
I’m mounting a campaign to get it into the ’stable’ branch that I’m using. In the meantime I can carry on working on stuff that I’d postponed until I got the leak sorted.
Last night/tonight I worked out how to do/use the following parts of Mozilla XPCOM
In addition to that I
I aim to finish by the end of the week.
Yesterday, regress-324422-1.js was checked into the Mozilla CVS repository (where the source code lives) by Bob Clary with the comment
Regression test for bug 324422, by Doug Wright
The important part of that file is line 23. Yay me!
(This was the first of the 2 bugs stopping progress on Companion 0.2, as mentioned here)
So far, not a lot – though not for lack of trying.
Companion 0.1 converts skins to XSL, applies the XSL to the XML, chops the result into lots of small pieces, and inserts them back into the skin (all using JavaScript). At the time this seemed like the best way of doing it – rapid prototyping, quick feedback when something was broken (get no output back), let me brush up on my XSL.
This approach also has drawbacks. The first one being the feedback when something doesn’t work as expected – there’s only one sort of error that gets returned which doesn’t help. When ‘new’ skin tags are processed, the XSL can’t handle it and dies. This is one of many situations I’d like to flag, so that an appropriate error message can displayed to the user. It also offers no flexibility in output – e.g. dates are always output in UK format, even for US users. Both of these (and others) could be included in the current setup, but with significant performance penalties, because each JS->XSL->JS roundtrip involves converting a JavaScript ’string’ into an XML ‘document’, and the resulting XML ‘document’ into a JavaScript ’string’. This conversion is the slowest part of the whole process, despite the fact it’s been optimised as far as possible (I process the entire XML file at once. The original design had a seperate pass for each profile, which kept RAM usage down, but took about 20x longer).
So I decided to rip out the XSL and take advantage of Mozilla’s support for E4X (EcmaScript for XML). E4X basically extends the JavaScript language to have native support for handling XML. Being a very new addition to the engine, it’s largely untested and my first line of E4X exposed two bugs in Mozilla’s implementation that have kept me back.
Bug #1 (Mozilla bug #324422): A crash when reading the XML into memory.
Bug #2 (Mozilla bug #324844): A limit of 4Mb on any native JS object.
The Mozilla devs are taking both issues seriously – the 1st one is already has a ‘fix’ and the 2nd is being worked on. Both are likely to be marked as ’stop-ship’ bugs for the next version.
Who said open source isn’t great?
I ’softly’ launched this last night, and made an announcement in the IVS forums this morning. I’ve only had 1 piece of feedback so far on this (from one of my early testers), and he seemed fairly happy…
I’m going to take a few days off now and start watching some DVDs again. Then I’ve got to make a decent user interface. That’ll require some thought.
Also (to add to this post a few days ago), I’ve discovered another ‘feature’ of IVS’s skin tags. The ‘=’ is optional…
Q: When will the DVD Profiler Companion get it’s initial public release.
A: When I track down (and fix) the cause of a bug that causes a crashes on a semi-regular basis.