Skip to content

Commit

Permalink
test_runner: force-compact in test_sharding_autosplit (#10605)
Browse files Browse the repository at this point in the history
## Problem

This test may not fully detect data corruption during splits, since we
don't force-compact the entire keyspace.

## Summary of changes

Force-compact all data in `test_sharding_autosplit`.
  • Loading branch information
erikgrinaker authored Jan 31, 2025
1 parent 7d5c70c commit afbcebe
Showing 1 changed file with 11 additions and 1 deletion.
12 changes: 11 additions & 1 deletion test_runner/performance/test_sharding_autosplit.py
Original file line number Diff line number Diff line change
Expand Up @@ -247,7 +247,7 @@ def assert_all_split():
log.info(f"{shard_zero_id} timeline: {timeline_info}")

# Run compaction for all tenants, restart endpoint so that on subsequent reads we will
# definitely hit pageserver for reads. This compaction passis expected to drop unwanted
# definitely hit pageserver for reads. This compaction pass is expected to drop unwanted
# layers but not do any rewrites (we're still in the same generation)
for tenant_id, tenant_state in tenants.items():
tenant_state.endpoint.stop()
Expand Down Expand Up @@ -296,6 +296,16 @@ def assert_all_split():
for fut in pgbench_futs:
fut.result()

# Run a full forced compaction, to detect any data corruption.
for tenant_id, tenant_state in tenants.items():
for shard_id, shard_ps in tenant_get_shards(env, tenant_id):
shard_ps.http_client().timeline_compact(
shard_id,
tenant_state.timeline_id,
force_image_layer_creation=True,
force_l0_compaction=True,
)

# Assert that some rewrites happened
# TODO: uncomment this after https://github.com/neondatabase/neon/pull/7531 is merged
# assert any(ps.log_contains(".*Rewriting layer after shard split.*") for ps in env.pageservers)

1 comment on commit afbcebe

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

7565 tests run: 7203 passed, 1 failed, 361 skipped (full report)


Failures on Postgres 16

# Run all failed tests locally:
scripts/pytest -vv -n $(nproc) -k "test_layer_map[release-pg16-github-actions-selfhosted]"

Code coverage* (full report)

  • functions: 33.4% (8517 of 25538 functions)
  • lines: 49.1% (71522 of 145571 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
afbcebe at 2025-01-31T14:30:23.181Z :recycle:

Please sign in to comment.