Benchmark test suite that allows easy comparison of competing implementations

[eriknw]

I know the performance of (possibly several) functions in CyToolz can be improved, but there is no good way to compare competing implementations against a common suite of benchmarks. If a contributor (myself included) wants to improve a function, they probably need to engage in time-consuming trial and error using their own benchmarks. I want to remedy this situation.

I envision the following:

• "bench/" directory in the repo root directory.
• One *.pyx file per function that is named after the function.
• Each *.pyx file can have several implementations of the function.
  • The name of each alternative implementation must begin with the name of the function, such as get1 or get_with_itemgetter.
  • This will allow for automated introspection.
• Timing will be automated with the timeit module.
• The tests are run by calling the file as main, such as python get.pyx.
• The files will be build automatically even when run as main.
• The output will be an ASCII table of benchmarks that can be copy/pasted as a github-flavored Markdown table.
  • Columns will be implementation names, including toolz, cytoolz, and suffixes of all alternative implementations.
  • Rows will be benchmarks names.
  • Additional rows may be added for aggregated statistics, such as harmonic mean.
  • Data will be absolute time or relative time of benchmarks (one table for each data type).

One advantage of doing this is to have a record of alternative implementations that have been tried. It also allows faster trial and error testing of implementations.

I would have loved to have something like this while first developing CyToolz. I have several benchmarks and alternative implementations haphazardly lying around, and it would be best for everybody to have them shared and usable in a sane way.

I plan to tackle this very soon, but I have not yet begun, so feedback and alternative ideas are most welcome!

[mrocklin]

If we had time then I can imagine making a small nosetests like utility.
I'll give this some thought.
On May 2, 2014 1:43 PM, "Erik Welch" notifications@github.com wrote:

I know the performance of (possibly several) functions in CyToolz can be
improved, but there is no good way to compare competing implementations
against a common suite of benchmarks. If a contributor (myself included)
wants to improve a function, they probably need to engage in time-consuming
trial and error using their own benchmarks. I want to remedy this situation.

I envision the following:

• "bench/" directory in the repo root directory.
• One *.pyx file per function that is named after the function.
• Each *.pyx file can have several implementations of the function.
  • The name of each alternative implementation must begin with the
    name of the function, such as get1 or get_with_itemgetter.
  • This will allow for automated introspection.
    • Timing will be automated with the timeit module.
• The tests are run by calling the file as main, such as python get.pyx
  .
• The files will be build automatically even when run as main.
• The output will be an ASCII table of benchmarks that can be
  copy/pasted as a github-flavored Markdown table.
  • Columns will be implementation names, including toolz, cytoolz,
    and suffixes of all alternative implementations.
  • Rows will be benchmarks names.
  • Additional rows may be added for aggregated statistics, such as
    harmonic mean.
  • Data will be absolute time or relative time of benchmarks (one
    table for each data type).

One advantage of doing this is to have a record of alternative
implementations that have been tried. It also allows faster trial and error
testing of implementations.

I would have loved to have something like this while first developing
CyToolz. I have several benchmarks and alternative implementations
haphazardly lying around, and it would be best for everybody to have them
shared and usable in a sane way.

I plan to tackle this very soon, but I have not yet begun, so feedback and
alternative ideas are most welcome!

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/22
.

[eriknw]

We should also consider using IPython notebooks.

[eriknw]

Interesting: https://github.com/simonpercivall/bettertimeit is a nosetest-like utility for timing functions. It doesn't have many of the features I want, but it is a very young package, so perhaps it is possible to help add these features.

In general, I prefer not to use IPython notebooks for things like this, and instead prefer utilities that are usable from the command line and allow code to be written in plain text files with my favorite editor, especially if the utilities are general and reusable.

[eriknw]

I'm now envisioning a benchtoolz repository in the PyToolz suite that, while tailored to our needs, is not specifically tied to toolz. We may want to look at https://github.com/pydata/vbench for ideas or as a starting point. Another repository such as benchmarkz (or something more generically named) could have the actual benchmarks.

I think this would be useful for both toolz and cytoolz. For example, a common optimization not found it toolz is avoiding attribute lookups in tight loops. Sometimes this may be worth it, and sometimes it may just make the code less readable. toolz.unique is a case where it is worth it: setting seen_add = seen.add results in a significant improvement.

While I don't want to dump a lot of time into this, I think something reasonably good can be whipped up pretty quickly.

[eriknw]

@mrocklin, are you okay with the name of the standalone package being benchtoolz? It could be named something else like benchmatrix. I don't mean to co-opt the *toolz moniker, but I do like benchtoolz as the name and would like your approval to use it and to add the package to PyToolz.

[mrocklin]

I'm fine with the name benchtoolz. I think it's a great idea so I'm happy to be accidentally associated to it :)

[eriknw]

I think I like the name benchit even better, which surprisingly wasn't taken. benchit is also better suited as a command line executable. Thoughts?

[mrocklin]

Sounds good.

On Wed, May 21, 2014 at 8:49 PM, Erik Welch notifications@github.comwrote:

I think I like the name benchit even better, which surprisingly wasn't
taken. benchit is also better suited as a command line executable.
Thoughts?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/22#issuecomment-43846010
.

[mrocklin]

I googled benchit because I was wondering if it was up on PyPI yet. You have competition for the name.

[eriknw]

What did you find with google? I have benchit reserved on PyPI.

[mrocklin]

Just lots of little benchmarking utilities for different systems. https://www.google.com/#q=benchit

It's not a big deal. PyToolz overlaps with PyTools, a visual studio thing. It makes googling and searching a bit more of a pain. The Z really helps.

[eriknw]

It's a good point though. I had googled "benchit python" originally, and the results were pretty weak.

Perhaps the Python package can be benchtoolz, and the command line utility--supposing there will be one eventually--can be benchit or some other name. I may be picky in that I don't like it when Python packages simultaneously install binaries too. Plus, command line utilities can become a hassle once you introduce custom configurations, so having this as a separate project would allow both of them to be as simple as possible.

[eriknw]

I just pushed benchtoolz: https://github.com/eriknw/benchtoolz

@mrocklin, shall we clone this into PyToolz? Do you have suggestions for a different name? Do you like the idea of having benchmarks for toolz and cytoolz defined in a separate repository (named something like "benchmarkz" or "pytoolz_benchmarks")?

A lot of work remains to make this a generally useful--and clean--package, but the README should illustrate the vision for the project. The README is currently too long for a readme (imho), but it should form the base of the online documentation. Right now, I would call benchtoolz a mostly functioning hack :) , and I think it works well enough that we can begin using it.

Feedback and contributions are most welcome. I would love to get somebody else interested in this project who can be a primary author.