Add notes about open, verified

swe-bench · Jun 21, 2024 · f1bae1e · f1bae1e
1 parent d84bdc0
commit f1bae1e
Show file tree

Hide file tree

Showing 2 changed files with 14 additions and 2 deletions.
diff --git a/index.html b/index.html
@@ -472,7 +472,13 @@ <h2 class="text-title">Leaderboard</h2>
               - The <span style="color:#0ea7ff;"><b>% Resolved</b></span> metric refers to the percentage of SWE-bench instances (2294 total)
               that were <i>resolved</i> by the model.
               <br>
-              - The leaderboard will be updated once a week on Monday.
+              - <span style="color:#0ea7ff;">"<b>Verified</b>"</span> indicates that we, the SWE-bench team, received access to the system and
+              were able to reproduce the patch generations.
+              <br>
+              - <span style="color:#0ea7ff;">"<b>Open</b>"</span> refers to submissions that have open-source code. This does <i>not</i>
+              necessarily mean the underlying model is open-source.
+              <br>
+              - The leaderboard is updated once a week on <b>Monday</b>.
               <br>
               - If you would like to submit your model to the leaderboard, please check the <a href="submit.html">submission</a> page.
               <br>

diff --git a/template/template.html b/template/template.html
@@ -162,7 +162,13 @@ <h2 class="text-title">Leaderboard</h2>
               - The <span style="color:#0ea7ff;"><b>% Resolved</b></span> metric refers to the percentage of SWE-bench instances (2294 total)
               that were <i>resolved</i> by the model.
               <br>
-              - The leaderboard will be updated once a week on Monday.
+              - <span style="color:#0ea7ff;">"<b>Verified</b>"</span> indicates that we, the SWE-bench team, received access to the system and
+              were able to reproduce the patch generations.
+              <br>
+              - <span style="color:#0ea7ff;">"<b>Open</b>"</span> refers to submissions that have open-source code. This does <i>not</i>
+              necessarily mean the underlying model is open-source.
+              <br>
+              - The leaderboard is updated once a week on <b>Monday</b>.
               <br>
               - If you would like to submit your model to the leaderboard, please check the <a href="submit.html">submission</a> page.
               <br>