Measuring DOM performance

[Boris Zbarsky]

We really need to get something measuring DOM performance running on
Tinderbox. Current plan is to run the updated dromaeo, but does this
actually measure what we want? If not, can we either get tests added to
it (or removed from it), or do we need a different solution altogether
long-term? Short-term dromaeo is of course better than nothing....

-Boris

P.S. Followups to .dom, please.

[Damon Sicore]

Post by Boris Zbarsky
We really need to get something measuring DOM performance running on
Tinderbox. Current plan is to run the updated dromaeo, but does
this actually measure what we want? If not, can we either get tests
added to it (or removed from it), or do we need a different solution
altogether long-term? Short-term dromaeo is of course better than
nothing....
-Boris

Boris, it sounds like you have a few things in mind. Maybe list them
out?

I think we really need to push on this. As far as I can tell--from
statements from others--we don't have visibility into DOM perf at all.

Post by Boris Zbarsky
P.S. Followups to .dom, please.
_______________________________________________
dev-platform mailing list
https://lists.mozilla.org/listinfo/dev-platform

[Boris Zbarsky]

Boris, it sounds like you have a few things in mind. Maybe list them out?

I don't offhand; otherwise I would have said what they are...

I haven't done enough digging into dromaeo yet to know what it does and
doesn't cover, basically.

I think we really need to push on this. As far as I can tell--from
statements from others--we don't have visibility into DOM perf at all.

Agreed.

-Boris

[John Resig]

Post by Boris Zbarsky
I haven't done enough digging into dromaeo yet to know what it does and
doesn't cover, basically.

It's a pretty good slice of the DOM API - but it doesn't cover
anything like rendering, canvas, CSS, etc.

It also includes a number of tests covering JavaScript libraries - so
you'll probably get a pretty good sample of library DOM perf at the
same time.

--John

[Boris Zbarsky]

Post by John Resig

Post by Boris Zbarsky
I haven't done enough digging into dromaeo yet to know what it does and
doesn't cover, basically.

It's a pretty good slice of the DOM API - but it doesn't cover
anything like rendering, canvas, CSS, etc.

Covers in what sense? It's one thing to test insertion of a single node
into a parent. It's another to test insertion of a subtree. Oh, and the
behavior might depend on the nodes being inserted (do they have names or
ids?), on the parent (is it display:none or not? is it actually in a
document?) and some other things.

That's all one single appendChild DOM API.

-Boris

[John Resig]

Covers in what sense?  It's one thing to test insertion of a single node
into a parent. It's another to test insertion of a subtree.  Oh, and the
behavior might depend on the nodes being inserted (do they have names or
ids?), on the parent (is it display:none or not?  is it actually in a
document?) and some other things.
That's all one single appendChild DOM API.

Covers in the sense that it uses the method for some form of testing
(the particulars will, naturally, need to be analyzed further). It
sounds like you're more interested in comprehensive regression tests -
whereas Dromaeo is more optimized towards analyzing the performance of
the most-commonly performed tasks (crossed by most DOM methods).

--John

[Boris Zbarsky]

Post by John Resig
Covers in the sense that it uses the method for some form of testing
(the particulars will, naturally, need to be analyzed further). It
sounds like you're more interested in comprehensive regression tests

I'm interested in several things, if I get to wish for ponies and rainbows:

1) Performance regression tests.
2) Tests that might indicate whether the DOM performance improvements
we think we're making actually have any effect on something other
than our tests.
3) Tests that might indicate where we should prioritize our work in
order to maximize impact on web content.

Post by John Resig
whereas Dromaeo is more optimized towards analyzing the performance of
the most-commonly performed tasks (crossed by most DOM methods).

I'm not entirely convinced. Maybe I have bad luck, but every time I've
dug into a dromaeo test in detail (admittedly only 4 different tests so
far, but digging in detail takes a while) I've found that it was gated
by weird things (substring() calls offset more than 16k characters into
a string, cross-window function calls inside tight loops, that sort of
thing) that I've seen nowhere outside dromaeo in the years I've been
dealing with browser performance. Said weird things, being weird, were
not highly optimized for. Now this is not to say the data was useless:
in at least one of the cases we did some measurements and decided that
the existing tuning was not reflecting web reality as well as it could;
a side benefit was speeding up the relevant dromaeo test by 5x or so.
But as Brendan keeps saying, we shouldn't be tuning to dromaeo; we
should be tuning to the web.

This is a general issue with performance tests, computer and otherwise:
tuning to the test. Since it's inevitable for whatever we run on
tinderbox, the goal should be to make sure that tuning to the test tunes
to whatever we actually care about (which is not the test per se).

In the short run, testing _something_ DOM-related on tinderbox is better
than not testing it at all, and dromaeo is here and ready to be used.
This thread is about the slightly longer run.

-Boris

[Jonas Sicking]

Post by John Resig

Post by Boris Zbarsky
Covers in what sense? It's one thing to test insertion of a single node
into a parent. It's another to test insertion of a subtree. Oh, and the
behavior might depend on the nodes being inserted (do they have names or
ids?), on the parent (is it display:none or not? is it actually in a
document?) and some other things.
That's all one single appendChild DOM API.

Covers in the sense that it uses the method for some form of testing
(the particulars will, naturally, need to be analyzed further). It
sounds like you're more interested in comprehensive regression tests -
whereas Dromaeo is more optimized towards analyzing the performance of
the most-commonly performed tasks (crossed by most DOM methods).

My main concern on the subject of "measureing DOM performance" is having
tests that measure *only* the DOM. Many times you end up testing the
performance of for example the layout engine as well. Any time you do
modifications in a document that is rendered layout code has to kick in.

Sometimes the layout code ends up not doing anything (for example if you
modify things inside a display:none node), however it can take some time
before we realize that nothing needs to be done.

/ Jonas

[Boris Zbarsky]

Post by Jonas Sicking
My main concern on the subject of "measureing DOM performance" is having
tests that measure *only* the DOM. Many times you end up testing the
performance of for example the layout engine as well. Any time you do
modifications in a document that is rendered layout code has to kick in.

I think we want tests like this too; possibly in addition to those we
want pure-DOM tests...

It should be possible to minimize the layout part by working in
disconnected subtrees or data documents, right?

-Boris

[Jonas Sicking]

Post by Boris Zbarsky

Post by Jonas Sicking
My main concern on the subject of "measureing DOM performance" is having
tests that measure *only* the DOM. Many times you end up testing the
performance of for example the layout engine as well. Any time you do
modifications in a document that is rendered layout code has to kick in.

I think we want tests like this too; possibly in addition to those we
want pure-DOM tests...
It should be possible to minimize the layout part by working in
disconnected subtrees or data documents, right?

True.

I guess the answer is simply that we should be testing what people are
doing. And note that modifying things in-displayed-document can be very
different performance-wise than out-of-displayed-document. So if people
do both, we should test both.

/ Jonas

[Boris Zbarsky]

Post by Jonas Sicking
I guess the answer is simply that we should be testing what people are
doing. And note that modifying things in-displayed-document can be very
different performance-wise than out-of-displayed-document. So if people
do both, we should test both.

Precisely my thoughts.

-Boris

[Jonas Sicking]

Post by Boris Zbarsky
It should be possible to minimize the layout part by working in
disconnected subtrees or data documents, right?

Yes, in these cases I think layout won't be notified at all. If we do
want to test pure-DOM performance than that is the way to do so.

/ Jonas

[Jean-Marc Desperrier]

Current plan is to run the updated dromaeo, but does this actually
measure what we want?

It obviously measures js performance in addition to DOM performance.
A first step is to disable jit to not measures the jit engine ability to
jit the dromaeo code, but then js performance variations might easily
hide the DOM performance variations.

Do you need to measure the performance of every DOM operation ? Of the
most commonly used ones on the web ? Is measuring a short set of simple
DOM operation a good enough proxy ?

[Boris Zbarsky]

Post by Jean-Marc Desperrier
It obviously measures js performance in addition to DOM performance.

Well, hold on. When we talk about "DOM performance" this means "DOM
performance as experienced by web pages". Which means including the
performance of the JS-to-C++ bridge, in particular.

Post by Jean-Marc Desperrier
A first step is to disable jit to not measures the jit engine ability to
jit the dromaeo code

This affects the JS-to-C++ bridge performance, so it's not clear to me
that this is the right thing to do.

Post by Jean-Marc Desperrier
Do you need to measure the performance of every DOM operation ?

Since the set of possible operations (meaning DOM method calls and
arguments and contexts for the calls) is infinite, this is not really
doable. I'd like us to not limit what we measure, though, which is
easier if we're not just measuring a single number.

Post by Jean-Marc Desperrier
Of the most commonly used ones on the web ?

This is a bare minimum, but do we even know what DOM operations are
commonly used? And the contexts they're used in?

Post by Jean-Marc Desperrier
Is measuring a short set of simple DOM operation a good enough proxy ?

As a start it might be; that's where I sort of view dromaeo as being
right now, if you ignore its js framework tests. The js framework tests
are closer to "do this operation with more different arguments and in
different contexts".

-Boris