WR vector image invalidation

jrmuizel's understanding of what chrome does:

• Has a list of threadsafe refcounted skpictures (one per display item)
• Builds a new list of these skpictures reusing them as necessary
• Copies the list over to the compositor
• The compositor paints it

Our plan for Firefox:

DLBI figures out the invalid region for the PaintedLayer. In DrawPaintedLayer, do not clip to the invalid region. Instead, iterate over all the display items that intersect the invalid region, and record each display item's drawing commands to an individual SkPicture. Retain that SkPicture on that display item's display item data. For the other display items on the layer (those that don't intersect the visible region), we already have retained SkPictures from a previous paint. As part of submitting that WebRenderPaintedLayer, merge the SkPictures of all display items that are assigned to this layer into a single SkPicture and send that merged SkPicture to webrender. This SkPicture lets you draw the whole layer. Also forward along the invalid region to webrender. On the webrender side, for each tile that touches the invalid region, re-execute the whole merged SkPicture. If necessary, webrender can clip to the invalid region if it has retained the pixels for the rest of the tile.

Having a merged SkPicture means that we can always redraw the whole layer, even as new parts of the layer come onscreen that we didn't have to paint before. Retaining SkPictures separately for each display item means that we don't need to re-record drawing commands for display items that don't intersect the invalid region. This way we can take advantage of DLBI on the content side.

Worries:

• Chrome is not serializing the SkPictures a lot because there compositor runs in the same thread (our approach might get us into trouble here. Though it looks like Chrome is headed there too with there new architecture)

question:

• will the commands in SkPicture become more and more? We retain the SkPicture from previous paint and plus the new CMD for the items in invalid region. (jrmuizel's feelings are that we'll have some mutable SkPicture that will change instead of accumulating new cmds. I believe that is what Chrome does)

Wouldn't each display item either retain its existing recording, or get a new one entirely? From there we always combine all display items into the combined recording for the layer from scratch, so the recording shouldn't grow.

April 11

The plan above assembles a complete SkPicture for all display items in the layer on the content side, on every paint. It then sends over that complete SkPicture. Can we avoid sending over the whole layer, and instead only send over SkPictures for display items that were changed? That means that the webrender side needs to keep track of individual SkPictures per display item, and it needs to have enough information in order to apply the changes to its list correctly. Below is a plan that achieves this:

Blob image merging

• The blob image contains items which have ids. The id can be (nsDisplayItem->Frame(), nsDisplayItem->GetPerFrameKey())
• When sending an update we send a list of invalid rects and a list of display items that need to be painted
• We then iterate over the two sorted lists of newly received display items
  • if a new item is not completely contained by the invalid rects that means it exists in the old image. We can use it to align the two lists. We scan forward to find it in the old image.
    • all items before the the new item that are not completely contained in the invalid rects are appended to the merged list.
    • if an old item is completely contained by invalid rects it is discarded
  • the remaining items that are not completely contained in the invalid rects are appended to the merged list

Potential problems: the intersection tests can be slow

• we can bounding box the invalidation rects to make it so that we only check individual rects if needed If we had a list of removed item ids, we could avoid the intersection test.

Recording:

• a Moz2D backend is added with a DrawTarget, sourceSurface, Path, font, etc. implementation
  • SourceSurface: holds an imageKey and hopefully reuse the SourceSurfaceSharedData stuff
  • Font: holds the font key and a font object
  • DrawTargetBlobRecording: serializes commands, keeps a list of all source surfaces it uses, a list of fonts etc.
    • If GetAsSurface is called, create a skia DT, replay itself in a buffer and return a DataSourceSurface
      • TODO: how does that interact with splitting/merging recording? Do we use a single recording DT for all items but have a way to segment its recording (in which case the DT can replay itself by gluing all recording segments somehow) or do each display item have a separate recording target and the replay merges them later (in which case it's not clear to me what it means to do GetAsSurface on one of the recording targets)

Replay:

• The ResourceCache provides a way for the BlobImageRenderer to get access to immutable views on images, fonts, etc through their keys (the keys are in the serialized recording)

July 6

• One blob image per DisplayItem
  • Lots of overdrawing? Lot's of memory usage?
• Collect all of the subelements of an SVG
• Collect all of the subelements of an SVG but stop when we get to something we understand like a transform or text
• FrameLayerBuild on the webrender side (collect blob images together and rasterize them into a combined target)
  • What do we need to do this well? coherency (some kind of consistency between display list updates)