Consider capturing text inside rectangle or point selections

[robertknight]

If a user uses the rectangle or point annotation tools we don't currently capture a quote selector, which means that you can't see that text in the sidebar or search for it.

It would be useful to capture that text in some form. This could be as a quote selector, or perhaps as part of a new field for alt text for the selected region.

How this would work depends on the shape:

• For a rectangle selection (or other shape with non-zero area), the text would be the characters inside the annotated region, up to some length limit, or perhaps the words that intersect the selection.
• For a point selection, the text could be the word (or span of a few words) that contains the point

[robertknight]

If we used the quote selector approach, there are a couple of issues to consider:

1. Anchoring code needs a policy to handle the case where an annotation has both a shape and a quote selector. For example it could always use the shape for anchoring and fall back to the quote if that fails to anchor.
2. If we do fall back to the quote, anchoring may not work because the text within the shape may not be continuous in the document's reading order.

[robertknight]

@mkdir-washington-edu recently shared https://www.w3.org/WAI/tutorials/images/decision-tree/ internally. When an image contains text, it recommends using that as the alt value of the image if the real text is not present nearby.