So I’ve got you.  This doesn’t make much sense, does it?  All except the oldest among you used a newline in your first program.  The tough problem is not re-displaying the page as it changes, that’s easy.  Okay, maybe not easy, but at least it’s already been solved.

Imagine that you’re typing an email to a lovely lady you want to move here from St. Petersburg.  You’ve finished a paragraph about naked scuba diving, you’ve told her about your pet rock collection and now it’s time to add you’re closing line.  “Sincerely yours” is a bit too formal, and “With all my love” might scare her off.  In any case, if you can’t hit enter to type that line, than she’s never going to move here and marry you.

Well, it can’t be too tough a problem, can it?  Olav Kjær wrote a great article about the problems and inconsistencies involved.  HTML is a great language for documents (I write software for the web, they make me say that), but its rules for containing text are pretty lax.  And when you give a user a mouse and allow him to just click anywhere on the page?  That’s just crazy talk!

Why do I care?  Well, I was editing a *cough* wiki page on OpenPlans.org using the Firefox 3 Beta (took me awhile to finish off this post, eh?).  When I clicked in the middle of the page and hit enter to start typing a new paragraph, half of my page disappeared.  Expected results?  Uh, a new paragraph?  We use Xinha, the open source WYSIWYG editor, and a pretty old version at that, but there was no problem in any other browser, or in previous versions of Firefox.

So I did what any self-respecting software engineer does when the problem’s not in his code, and he can’t understand it. I blamed someone else.  I was so worried about supporting the problem for the life of Firefox 3 that I even called John Resig, Mozilla’s Javascript Evangelist.  You’ll notice that he’s removed the phone number from his site (sorry John).

After filing the bug, I started to search through snapshots of Minefield (the testing and development version of Firefox), and was able to narrow it down to one commit.  Looking at the source (in nsRange.cpp), it turns out that the change was a bugfix that caused Firefox to correctly implement the W3C Range standard.  Before the fix, trying to create a backwards selection raised an exception, and after the fix it returned an empty selection, as it was supposed to.  That meant the Xinha code depended on broken behavior; and that I had work to do.

My first job was to find out what was wrong,  That’s easy, I have access to the code, I just have to find out where things went wrong.  Eeek.  “processRng” and “processSide”.  Well, it’s pretty obvious what those two functions do.  The first processes a range, and the second processes a side.  Thanks to some helpful comments, I know that it returns a neighbor node, and insertion type, and a “roam”. What that means?  No idea.  My favorite comment?

“I do not profess to understand any of this, simply applying a patch that others say is good — ticket:446”

After stepping through the code I was finally able to figure out what it was trying to do.  It divided the document into two pieces.  It cuts out everything from the current cursor to the end of the document, inserts a break, and then pastes it back in again.  It sounds like a simple enough idea, but I couldn’t for the life of me figure out what was going wrong.  So again I did what any self-respecting software engineer would do.  I decided to rewrite the algorithm from scratch.

It’s now six months later, and I’ve finally nailed this bug.  Of course, other things happened in between, but that’s always the case.  Let’s take a look at what’s so tough about newlines.

1. The first difficult problem is determining user intent.  If the user finishes typing a heading and then hits enter, they probably want to start typing text in a new paragraph.  If however, the cursor is in the middle of two sentences in that heading, they probably want to split it into two headings.  In a table cell and they probably just want a line break.  If they’re editing a definition list, they might want to insert a new definition, a new term, or even split two sentences into two seperate definitions or terms.
2. The second problem is cursor position.  Since a cursor is defined as a pointer to a node and an offset, in the following HTML snippet, the position just before the letter ‘T’ can be targeted with two different cursors.
   

   <p><em>Text</em></p>

   The first would be a pointer to the “<em>” element with an offset of zero.  This would mean we were pointed at the text node.  The second would be a pointer to the text node with a zero offset.  In this case, we are pointing at the characters of text, and not at a node.
3. Third is what it means to break a line.  In a list, breaking a line means creating a new list item.  In a preformatted block, it means a newline character.  In a table cell, you want a “<br>” element, and in a paragraph you want a new paragraph.  I won’t even get into how this changes for shift-enter.
4. The final tough problem is inline elements.  The formatting of the text at a given cursor position is the result of a tree of inline elements that heads up towards the containing block.  When splitting that block, you have to create a duplicate of this tree with all of the same elements, and you have to split each inline element into the parts that come before the cursor, and the parts that come after.

After having finished the majority of this back when I found the bug, I shelved the code and moved on to other things.  With the help of my colleague, Nicholas Bergson-Shilcock, I’ve picked this up again and finished it off.  This means that the new Phoenix Release (0.96) of Xinha will get a bugfix that makes Firefox 3 usable again.

All of the code for this fix is pluggable, and should be usable by anyone needing to break lines in HTML.  The only dependence is on W3C Ranges and DOM Selections.  Luckily, there’s been talk of a cross-platform W3C Range and DOM Selection library.

When the guys over at 37signals released their own super-light-weight WYSIWYG editor WysiHat, they talked about wanting to help with the problem.  Mozile, the Mozilla Inline Editor, actually has one, but it’s too tied to the editor to be able to useful elsewhere.  TinyMCE goes the other way and has an IE TextRange implementation for Firefox, and I’ve recently been told that FCKEditor has the beginnings of a usable library.  I’ve implemented the tough parts twice now (finding the DOM node and offset of the ranges start and end points) and learned the best way to do it.  For the next release of Xinha (0.97) I hope to bring my work together with the work of all other interested parties and release it as a library.  When we do that, users will finally be able to go back and forth between browsers and not have to fight to edit a document.

Until then…

Just last week, I was working on the new version of Xinha.  If you don't know, Xinha's a web-based document editor.  Embed it in your blog, your web software, so that you and your users can create web documents. Xinha is WYSIWYG, so there's no need to know HTML.  The Open Planning Project, my employer, uses Xinha to power OpenPlans, which is why I get to work on it.  Xinha is Open Source Software, so we use it, and contribute fixes and enhancements back to the original project.

I was working with Nicholas Bergson-Shilcock, my colleague, on his new plugin for Xinha.  With this plugin, you can finally make great footnotes in your documents.  We were testing his code on Internet Explorer, and we noticed IE acting strange.  Now I don't mean normal IE strange, IE is the bane of all web developers, so I'm used to strange.  (If you use IE, then please don't.  I don't care whether you use Mozilla Firefox, Google Chrome, Opera, Apple Safari, or if you connect to web servers directly with telnet.  Just do all web developers a favor and stop using IE.)

When I say strange, I mean screwy.  Certain places in the document just didn't seem to exist.  His code used Xinha in different ways than the rest of the plugins, so we were expecting edge cases.  But black holes?  Nobody expects black holes!

Editable documents are still the wild west of web development, and so I shouldn't be surprised.  Javascript and DOM has its Wyatt Earp and Doc Holliday, but document editing is too new to have seen the same kind of law enforcement.  When it comes to selection, manipulation, and document processing, the browser differences aren't well defined, and there are no libraries to abstract the problems.  Even Peter-Paul Koch (of QuirksMode) told me that "IE's TextRange is a disaster" when I asked for help.

After a bit of exploring the problem we figured out exactly what happens.  In Internet Explorer, you can't select the end of a text node (in javascript) if it's followed by a block node.  That means that for the valid HTML snippet:

<div>
  This is my first line
  <p>This is my second line</p>
</div>

You can't touch the end of the first line.  Let me say that again, you can't touch the end of the first line. What does that mean?  All of you DOM jockeys know how to get a reference to the node, and could manipulate the elements, but that's no help for the user.

Your user pushes that cursor beyond the event horizon.  They click on your footnote button to bring up a dialog.  You insert the text they type, and BAM!  The cursor's not where the user left it; you've just crapped markup at some other place in the document.  When you do things like that, users start to fear pressing buttons, and we can't have that.

Why haven't we seen it before?  Xinha was using pop-ups for dialogs, and they don't change the original selection.  Now that we've moved to a lightbox-style dialog system, we're moving the cursor about on the page, and we don't have a way to move it back.

How do we fix it?  Our first step was to test in IE8 beta to see if it was fixed.  No such luck; sometimes I wonder why I'm an optimist. ;-)  My next step was to try out StackOverflow, the new Jeff Atwood / Joel Spolsky software development community.  It's pretty hot right now, so I thought it would be a good place to get help, but again, no go.  The only answer I got was someone who seemed to remember some comments related to this bug in Javascript.  I tried to find the software he was referring to, but no bugfix there.  FCKeditor doesn't have a fix.  Neither does TinyMCE. Wikipedia offered up this link to a list of 5000 web-based editors.  I tried them all, and all of the software not using pop-ups had the exact same bug.

So, what can we do?  Unfortunately, I tried to see if there was a way to trick IE into moving the selection to where we want.  I tried moving the selection left, or right, and then back again.  I tried inserting content, then deleting it, but there was no direct way to solve the problem.  We ended up with three different workarounds, all of which have drawbacks, but are better than no solution at all:

Change the justification

If you change the justification on the current selection, IE modifies the document so that the selection continues to work.  Set it to no justification, and you even get valid HTML! Unfortunately, it re-parents the following element, moving it one node closer to the root of the document.

Insert an empty span

This works by making sure that you are attempting to select the span element, rather than a text node, and element selection actually works in IE.  It craps spans all over the document, though. and even though we try to clean these up, you never know.

Insert a visual cue

The final method works by inserting a visual cue for the user in the form of a little block (□), then selecting it.  If we're about to modify the document, or the user begins to type, the block will be removed automatically.  In any other case, the user will see the block and naturally want to delete it from the text.

All three are written in to the code, but we decided to default to the visual cue, because it's the safest in terms of damaging the markup.  Otherwise, we've done everything we could to avoid triggering the error, so we hope it won't affect too many users; it's always a trade off.

I wrote this to get some visibility for this problem.  This is probably just some sort of off by one error, and IE8 is still in beta, so maybe it can still get fixed.  If not, at least you'll have a way to work around the problem when you run into it.