One thing the Nouveau project is doing well is to make open-source graphics more accessible. There are links to a bunch of articles which while still young give insights into what does what. Looking at the nouveau source code shows it really isn't overly complicated - that hard work has already been done by Mesa. A DRI 3D driver basically requires a kernel module to do the IO and an X driver that can convert OpenGL to native card protocol. The difficulty is this protocol is not (publically) documented. But looking through the code and googling shows that a lot of the information is out there. It needs to be compiled into a "Missing Manual".
If I had the time I would love to work on a project like this. Once past a critical point the value to the open-source world would be immense.
As Dave Airlie and Ben Skeggs said at LinuxConf.au; "Graphics drivers are not that hard..."