Skip to content

feat(auto-edit): fix the temperature value regression with the auto-e… #26137

feat(auto-edit): fix the temperature value regression with the auto-e…

feat(auto-edit): fix the temperature value regression with the auto-e… #26137

Triggered via push January 28, 2025 16:13
Status Failure
Total duration 18m 23s
Artifacts 1

ci.yml

on: push
fast_tests_matrix_prep
0s
fast_tests_matrix_prep
slow_tests_matrix_prep
0s
slow_tests_matrix_prep
Matrix: test-integration
Matrix: test-unit
Matrix: test-e2e
Fit to window
Zoom out
Zoom in

Annotations

10 errors, 15 warnings, and 3 notices
test-e2e (macos)
Worker teardown timeout of 30000ms exceeded. Failed worker ran 9 tests: test/e2e/chat-atFile.test.ts:229:3 › editing a chat message with @-mention test/e2e/chat-atFile.test.ts:271:3 › @-mention file range test/e2e/chat-atFile.test.ts:310:3 › @-mention symbol in chat test/e2e/chat-atFile.test.ts:367:3 › Add Selection to Cody Chat test/e2e/chat-history.test.ts:17:3 › restore chat from sidebar history view test/e2e/chat-history.test.ts:50:3 › delete chat from sidebar history view test/e2e/chat-input.test.ts:191:3 › chat model selector test/e2e/chat-input.test.ts:242:3 › editing follow-up messages in chat view test/e2e/chat-messages.test.ts:17:5 › chat assistant response code buttons
test-e2e (macos)
Worker teardown timeout of 30000ms exceeded. Failed worker ran 1 test: test/e2e/chat-messages.test.ts:17:5 › chat assistant response code buttons
auto-edits.test.ts:182:3 › autoedits: triggers a multi-line diff view when edit affects existing lines: vscode/test/e2e/auto-edits.test.ts#L160
1) auto-edits.test.ts:182:3 › autoedits: triggers a multi-line diff view when edit affects existing lines Error: Screenshot comparison failed: 44076 pixels (ratio 0.03 of all image pixels) are different. Expected: /Users/runner/work/cody/cody/vscode/test/e2e/__snapshots__/auto-edits.test.ts/macos/autoedits-suffix-decoration-70-suggested.png Received: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-1c59f-edit-affects-existing-lines/macos/autoedits-suffix-decoration-70-suggested-actual.png Diff: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-1c59f-edit-affects-existing-lines/macos/autoedits-suffix-decoration-70-suggested-diff.png Call log: - expect.toHaveScreenshot(macos/autoedits-suffix-decoration-70-suggested.png) with timeout 5000ms - verifying given screenshot expectation - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - 44077 pixels (ratio 0.03 of all image pixels) are different. - waiting 100ms before taking screenshot - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - captured a stable screenshot - 44076 pixels (ratio 0.03 of all image pixels) are different. 158 | await page.waitForTimeout(500) 159 | > 160 | await expect(page).toHaveScreenshot( | ^ 161 | [snapshotPlatform, `${snapshotName}-suggested.png`], 162 | SNAPSHOT_ASSERTIONS 163 | ) at autoeditsTestHelper (/Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:160:28) at /Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:186:9
auto-edits.test.ts:182:3 › autoedits: triggers a multi-line diff view when edit affects existing lines: vscode/test/e2e/auto-edits.test.ts#L160
1) auto-edits.test.ts:182:3 › autoedits: triggers a multi-line diff view when edit affects existing lines Retry #1 ─────────────────────────────────────────────────────────────────────────────────────── Error: Screenshot comparison failed: 44076 pixels (ratio 0.03 of all image pixels) are different. Expected: /Users/runner/work/cody/cody/vscode/test/e2e/__snapshots__/auto-edits.test.ts/macos/autoedits-suffix-decoration-70-suggested.png Received: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-1c59f-edit-affects-existing-lines-retry1/macos/autoedits-suffix-decoration-70-suggested-actual.png Diff: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-1c59f-edit-affects-existing-lines-retry1/macos/autoedits-suffix-decoration-70-suggested-diff.png Call log: - expect.toHaveScreenshot(macos/autoedits-suffix-decoration-70-suggested.png) with timeout 5000ms - verifying given screenshot expectation - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - 44077 pixels (ratio 0.03 of all image pixels) are different. - waiting 100ms before taking screenshot - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - captured a stable screenshot - 44076 pixels (ratio 0.03 of all image pixels) are different. 158 | await page.waitForTimeout(500) 159 | > 160 | await expect(page).toHaveScreenshot( | ^ 161 | [snapshotPlatform, `${snapshotName}-suggested.png`], 162 | SNAPSHOT_ASSERTIONS 163 | ) at autoeditsTestHelper (/Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:160:28) at /Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:186:9
auto-edits.test.ts:198:3 › autoedits: triggers an inline completion when edit is an insertion immediately after the cursor: vscode/test/e2e/auto-edits.test.ts#L160
2) auto-edits.test.ts:198:3 › autoedits: triggers an inline completion when edit is an insertion immediately after the cursor Error: Screenshot comparison failed: 20127 pixels (ratio 0.01 of all image pixels) are different. Expected: /Users/runner/work/cody/cody/vscode/test/e2e/__snapshots__/auto-edits.test.ts/macos/autoedits-inline-completion-29-suggested.png Received: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-912dc-mmediately-after-the-cursor/macos/autoedits-inline-completion-29-suggested-actual.png Diff: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-912dc-mmediately-after-the-cursor/macos/autoedits-inline-completion-29-suggested-diff.png Call log: - expect.toHaveScreenshot(macos/autoedits-inline-completion-29-suggested.png) with timeout 5000ms - verifying given screenshot expectation - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - 28719 pixels (ratio 0.02 of all image pixels) are different. - waiting 100ms before taking screenshot - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - 8628 pixels (ratio 0.01 of all image pixels) are different. - waiting 250ms before taking screenshot - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - captured a stable screenshot - 20127 pixels (ratio 0.01 of all image pixels) are different. 158 | await page.waitForTimeout(500) 159 | > 160 | await expect(page).toHaveScreenshot( | ^ 161 | [snapshotPlatform, `${snapshotName}-suggested.png`], 162 | SNAPSHOT_ASSERTIONS 163 | ) at autoeditsTestHelper (/Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:160:28) at /Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:202:9
auto-edits.test.ts:198:3 › autoedits: triggers an inline completion when edit is an insertion immediately after the cursor: vscode/test/e2e/auto-edits.test.ts#L160
2) auto-edits.test.ts:198:3 › autoedits: triggers an inline completion when edit is an insertion immediately after the cursor Retry #1 ─────────────────────────────────────────────────────────────────────────────────────── Error: Screenshot comparison failed: 20127 pixels (ratio 0.01 of all image pixels) are different. Expected: /Users/runner/work/cody/cody/vscode/test/e2e/__snapshots__/auto-edits.test.ts/macos/autoedits-inline-completion-29-suggested.png Received: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-912dc-mmediately-after-the-cursor-retry1/macos/autoedits-inline-completion-29-suggested-actual.png Diff: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-912dc-mmediately-after-the-cursor-retry1/macos/autoedits-inline-completion-29-suggested-diff.png Call log: - expect.toHaveScreenshot(macos/autoedits-inline-completion-29-suggested.png) with timeout 5000ms - verifying given screenshot expectation - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - 28719 pixels (ratio 0.02 of all image pixels) are different. - waiting 100ms before taking screenshot - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - 8628 pixels (ratio 0.01 of all image pixels) are different. - waiting 250ms before taking screenshot - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - captured a stable screenshot - 20127 pixels (ratio 0.01 of all image pixels) are different. 158 | await page.waitForTimeout(500) 159 | > 160 | await expect(page).toHaveScreenshot( | ^ 161 | [snapshotPlatform, `${snapshotName}-suggested.png`], 162 | SNAPSHOT_ASSERTIONS 163 | ) at autoeditsTestHelper (/Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:160:28) at /Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:202:9
auto-edits.test.ts:214:3 › autoedits: triggers an inline decoration when an inline completion is desired: vscode/test/e2e/auto-edits.test.ts#L160
3) auto-edits.test.ts:214:3 › autoedits: triggers an inline decoration when an inline completion is desired, but the insertion position is before the cursor position Error: Screenshot comparison failed: 26568 pixels (ratio 0.02 of all image pixels) are different. Expected: /Users/runner/work/cody/cody/vscode/test/e2e/__snapshots__/auto-edits.test.ts/macos/autoedits-inline-decoration-insertion-30-suggested.png Received: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-c9082--before-the-cursor-position/macos/autoedits-inline-decoration-insertion-30-suggested-actual.png Diff: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-c9082--before-the-cursor-position/macos/autoedits-inline-decoration-insertion-30-suggested-diff.png Call log: - expect.toHaveScreenshot(macos/autoedits-inline-decoration-insertion-30-suggested.png) with timeout 5000ms - verifying given screenshot expectation - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - 26550 pixels (ratio 0.02 of all image pixels) are different. - waiting 100ms before taking screenshot - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - captured a stable screenshot - 26568 pixels (ratio 0.02 of all image pixels) are different. 158 | await page.waitForTimeout(500) 159 | > 160 | await expect(page).toHaveScreenshot( | ^ 161 | [snapshotPlatform, `${snapshotName}-suggested.png`], 162 | SNAPSHOT_ASSERTIONS 163 | ) at autoeditsTestHelper (/Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:160:28) at /Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:218:9
auto-edits.test.ts:214:3 › autoedits: triggers an inline decoration when an inline completion is desired: vscode/test/e2e/auto-edits.test.ts#L160
3) auto-edits.test.ts:214:3 › autoedits: triggers an inline decoration when an inline completion is desired, but the insertion position is before the cursor position Retry #1 ─────────────────────────────────────────────────────────────────────────────────────── Error: Screenshot comparison failed: 26550 pixels (ratio 0.02 of all image pixels) are different. Expected: /Users/runner/work/cody/cody/vscode/test/e2e/__snapshots__/auto-edits.test.ts/macos/autoedits-inline-decoration-insertion-30-suggested.png Received: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-c9082--before-the-cursor-position-retry1/macos/autoedits-inline-decoration-insertion-30-suggested-actual.png Diff: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-c9082--before-the-cursor-position-retry1/macos/autoedits-inline-decoration-insertion-30-suggested-diff.png Call log: - expect.toHaveScreenshot(macos/autoedits-inline-decoration-insertion-30-suggested.png) with timeout 5000ms - verifying given screenshot expectation - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - 26550 pixels (ratio 0.02 of all image pixels) are different. - waiting 100ms before taking screenshot - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - captured a stable screenshot - 26550 pixels (ratio 0.02 of all image pixels) are different. 158 | await page.waitForTimeout(500) 159 | > 160 | await expect(page).toHaveScreenshot( | ^ 161 | [snapshotPlatform, `${snapshotName}-suggested.png`], 162 | SNAPSHOT_ASSERTIONS 163 | ) at autoeditsTestHelper (/Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:160:28) at /Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:218:9
auto-edits.test.ts:230:3 › autoedits: triggers inline decorations when multiple insertions are required on different lines: vscode/test/e2e/auto-edits.test.ts#L160
4) auto-edits.test.ts:230:3 › autoedits: triggers inline decorations when multiple insertions are required on different lines Error: Screenshot comparison failed: 23241 pixels (ratio 0.02 of all image pixels) are different. Expected: /Users/runner/work/cody/cody/vscode/test/e2e/__snapshots__/auto-edits.test.ts/macos/autoedits-inline-decoration-multiple-insertions-different-lines-44-suggested.png Received: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-d9853-required-on-different-lines/macos/autoedits-inline-decoration-multiple-insertions-different-lines-44-suggested-actual.png Diff: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-d9853-required-on-different-lines/macos/autoedits-inline-decoration-multiple-insertions-different-lines-44-suggested-diff.png Call log: - expect.toHaveScreenshot(macos/autoedits-inline-decoration-multiple-insertions-different-lines-44-suggested.png) with timeout 5000ms - verifying given screenshot expectation - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - 23240 pixels (ratio 0.02 of all image pixels) are different. - waiting 100ms before taking screenshot - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - captured a stable screenshot - 23241 pixels (ratio 0.02 of all image pixels) are different. 158 | await page.waitForTimeout(500) 159 | > 160 | await expect(page).toHaveScreenshot( | ^ 161 | [snapshotPlatform, `${snapshotName}-suggested.png`], 162 | SNAPSHOT_ASSERTIONS 163 | ) at autoeditsTestHelper (/Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:160:28) at /Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:234:9
auto-edits.test.ts:230:3 › autoedits: triggers inline decorations when multiple insertions are required on different lines: vscode/test/e2e/auto-edits.test.ts#L160
4) auto-edits.test.ts:230:3 › autoedits: triggers inline decorations when multiple insertions are required on different lines Retry #1 ─────────────────────────────────────────────────────────────────────────────────────── Error: Screenshot comparison failed: 23240 pixels (ratio 0.02 of all image pixels) are different. Expected: /Users/runner/work/cody/cody/vscode/test/e2e/__snapshots__/auto-edits.test.ts/macos/autoedits-inline-decoration-multiple-insertions-different-lines-44-suggested.png Received: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-d9853-required-on-different-lines-retry1/macos/autoedits-inline-decoration-multiple-insertions-different-lines-44-suggested-actual.png Diff: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-d9853-required-on-different-lines-retry1/macos/autoedits-inline-decoration-multiple-insertions-different-lines-44-suggested-diff.png Call log: - expect.toHaveScreenshot(macos/autoedits-inline-decoration-multiple-insertions-different-lines-44-suggested.png) with timeout 5000ms - verifying given screenshot expectation - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - 23240 pixels (ratio 0.02 of all image pixels) are different. - waiting 100ms before taking screenshot - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - captured a stable screenshot - 23240 pixels (ratio 0.02 of all image pixels) are different. 158 | await page.waitForTimeout(500) 159 | > 160 | await expect(page).toHaveScreenshot( | ^ 161 | [snapshotPlatform, `${snapshotName}-suggested.png`], 162 | SNAPSHOT_ASSERTIONS 163 | ) at autoeditsTestHelper (/Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:160:28) at /Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:234:9
Slow Test: vscode/chat-atFile.test.ts#L1
vscode/chat-atFile.test.ts took 1.4m
Slow Test: vscode/command-custom.test.ts#L1
vscode/command-custom.test.ts took 1.2m
Slow Test: vscode/initial-context.test.ts#L1
vscode/initial-context.test.ts took 54.3s
Slow Test: vscode/command-core.test.ts#L1
vscode/command-core.test.ts took 46.6s
Slow Test: vscode/command-edit.test.ts#L1
vscode/command-edit.test.ts took 44.8s
Slow Test: vscode/command-custom.test.ts#L1
vscode/command-custom.test.ts took 1.1m
Slow Test: vscode/initial-context.test.ts#L1
vscode/initial-context.test.ts took 47.6s
Slow Test: vscode/command-edit.test.ts#L1
vscode/command-edit.test.ts took 38.0s
Slow Test: vscode/command-core.test.ts#L1
vscode/command-core.test.ts took 37.1s
Slow Test: vscode/chat-rateLimit.test.ts#L1
vscode/chat-rateLimit.test.ts took 31.0s
Slow Test: vscode\chat-atFile.test.ts#L1
vscode\chat-atFile.test.ts took 1.6m
Slow Test: vscode\command-custom.test.ts#L1
vscode\command-custom.test.ts took 1.4m
Slow Test: vscode\initial-context.test.ts#L1
vscode\initial-context.test.ts took 1.1m
Slow Test: vscode\command-core.test.ts#L1
vscode\command-core.test.ts took 54.2s
Slow Test: vscode\command-edit.test.ts#L1
vscode\command-edit.test.ts took 51.2s
🎭 Playwright Run Summary
17 skipped 41 passed (10.2m)
🎭 Playwright Run Summary
8 failed auto-edits.test.ts:182:3 › autoedits: triggers a multi-line diff view when edit affects existing lines auto-edits.test.ts:198:3 › autoedits: triggers an inline completion when edit is an insertion immediately after the cursor auto-edits.test.ts:214:3 › autoedits: triggers an inline decoration when an inline completion is desired, but the insertion position is before the cursor position auto-edits.test.ts:230:3 › autoedits: triggers inline decorations when multiple insertions are required on different lines auto-edits.test.ts:246:3 › autoedits: triggers inline decorations when multiple separate insertions are required on the same line auto-edits.test.ts:260:1 › autoedits: triggers a suffix decoration and renders correctly in files that use tab based indentation auto-edits.test.ts:274:1 › autoedits: does not show any suggestion if the suffix decoration spans further than the end of the file chat-messages.test.ts:17:5 › chat assistant response code buttons ────────────────────────────── 1 flaky chat-atFile.test.ts:229:3 › editing a chat message with @-mention ────────────────────────────── 10 skipped 39 passed (15.1m)
🎭 Playwright Run Summary
17 skipped 41 passed (11.8m)

Artifacts

Produced during runtime
Name Size
playwright-recordings macos
13.2 MB