Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Share memory by copying. #180

Merged
merged 6 commits into from
Oct 31, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions .github/workflows/cross.sh
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,13 @@ echo aix ; GOOS=aix GOARCH=ppc64 go build .
echo js ; GOOS=js GOARCH=wasm go build .
echo wasip1 ; GOOS=wasip1 GOARCH=wasm go build .
echo linux-flock ; GOOS=linux GOARCH=amd64 go build -tags sqlite3_flock .
echo linux-noshm ; GOOS=linux GOARCH=amd64 go build -tags sqlite3_noshm .
echo linux-dotlk ; GOOS=linux GOARCH=amd64 go build -tags sqlite3_dotlk .
echo linux-nosys ; GOOS=linux GOARCH=amd64 go build -tags sqlite3_nosys .
echo darwin-flock ; GOOS=darwin GOARCH=amd64 go build -tags sqlite3_flock .
echo darwin-noshm ; GOOS=darwin GOARCH=amd64 go build -tags sqlite3_noshm .
echo darwin-dotlk ; GOOS=darwin GOARCH=amd64 go build -tags sqlite3_dotlk .
echo darwin-nosys ; GOOS=darwin GOARCH=amd64 go build -tags sqlite3_nosys .
echo windows-dotlk ; GOOS=windows GOARCH=amd64 go build -tags sqlite3_dotlk .
echo windows-nosys ; GOOS=windows GOARCH=amd64 go build -tags sqlite3_nosys .
echo freebsd-dotlk ; GOOS=freebsd GOARCH=amd64 go build -tags sqlite3_dotlk .
echo freebsd-nosys ; GOOS=freebsd GOARCH=amd64 go build -tags sqlite3_nosys .
echo solaris-flock ; GOOS=solaris GOARCH=amd64 go build -tags sqlite3_flock .
echo solaris-dotlk ; GOOS=solaris GOARCH=amd64 go build -tags sqlite3_dotlk .
4 changes: 0 additions & 4 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -60,10 +60,6 @@ jobs:
run: go test -v -tags sqlite3_dotlk ./...
if: matrix.os == 'macos-latest'

- name: Test no shared memory
run: go test -v -tags sqlite3_noshm ./...
if: matrix.os == 'ubuntu-latest'

- name: Test no locks
run: go test -v -tags sqlite3_nosys ./...
if: matrix.os == 'ubuntu-latest'
Expand Down
2 changes: 1 addition & 1 deletion driver/example_test.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
//go:build (linux || darwin || windows || freebsd || openbsd || netbsd || dragonfly || illumos || sqlite3_flock) && !sqlite3_nosys
//go:build ((linux || darwin || windows || freebsd || openbsd || netbsd || dragonfly || illumos) && !sqlite3_nosys) || sqlite3_flock || sqlite3_dotlk

package driver_test

Expand Down
2 changes: 1 addition & 1 deletion internal/util/mmap.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
//go:build unix && !(sqlite3_noshm || sqlite3_nosys)
//go:build unix && !sqlite3_nosys

package util

Expand Down
2 changes: 1 addition & 1 deletion internal/util/mmap_other.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
//go:build !unix || sqlite3_noshm || sqlite3_nosys
//go:build !unix || sqlite3_nosys

package util

Expand Down
2 changes: 1 addition & 1 deletion tests/bradfitz/sql_test.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
//go:build (linux || darwin || windows || freebsd || openbsd || netbsd || dragonfly || illumos || sqlite3_flock) && !sqlite3_nosys
//go:build ((linux || darwin || windows || freebsd || openbsd || netbsd || dragonfly || illumos) && !sqlite3_nosys) || sqlite3_flock || sqlite3_dotlk

package bradfitz

Expand Down
14 changes: 7 additions & 7 deletions vfs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,7 @@ The main differences are [file locking](#file-locking) and [WAL mode](#write-ahe

POSIX advisory locks, which SQLite uses on Unix, are
[broken by design](https://github.com/sqlite/sqlite/blob/b74eb0/src/os_unix.c#L1073-L1161).

On Linux and macOS, this package uses
Instead, on Linux and macOS, this package uses
[OFD locks](https://www.gnu.org/software/libc/manual/html_node/Open-File-Description-Locks.html)
to synchronize access to database files.

Expand Down Expand Up @@ -45,7 +44,7 @@ to check if your build supports file locking.

### Write-Ahead Logging

On little-endian Unix, this package uses `mmap` to implement
On Unix, this package may use `mmap` to implement
[shared-memory for the WAL-index](https://sqlite.org/wal.html#implementation_of_shared_memory_for_the_wal_index),
like SQLite.

Expand All @@ -54,6 +53,9 @@ a WAL database can only be accessed by a single proccess.
Other processes that attempt to access a database locked with BSD locks,
will fail with the [`SQLITE_PROTOCOL`](https://sqlite.org/rescode.html#protocol) error code.

You can also opt into a cross platform, in-process, memory sharing implementation
with the `sqlite3_dotlk` build tag.

Otherwise, [WAL support is limited](https://sqlite.org/wal.html#noshm),
and `EXCLUSIVE` locking mode must be set to create, read, and write WAL databases.
To use `EXCLUSIVE` locking mode with the
Expand All @@ -66,7 +68,7 @@ to check if your build supports shared memory.

### Batch-Atomic Write

On 64-bit Linux, this package supports
On Linux, this package may support
[batch-atomic writes](https://sqlite.org/cgi/src/technote/714)
on the F2FS filesystem.

Expand All @@ -87,9 +89,7 @@ The implementation is compatible with SQLite's
The VFS can be customized with a few build tags:
- `sqlite3_flock` forces the use of BSD locks.
- `sqlite3_dotlk` forces the use of dot-file locks.
- `sqlite3_nosys` prevents importing [`x/sys`](https://pkg.go.dev/golang.org/x/sys);
disables locking _and_ shared memory on all platforms.
- `sqlite3_noshm` disables shared memory on all platforms.
- `sqlite3_nosys` prevents importing [`x/sys`](https://pkg.go.dev/golang.org/x/sys).

> [!IMPORTANT]
> The default configuration of this package is compatible with the standard
Expand Down
2 changes: 1 addition & 1 deletion vfs/adiantum/example_test.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
//go:build (linux || darwin || windows || freebsd || openbsd || netbsd || dragonfly || illumos || sqlite3_flock) && !sqlite3_nosys
//go:build ((linux || darwin || windows || freebsd || openbsd || netbsd || dragonfly || illumos) && !sqlite3_nosys) || sqlite3_flock || sqlite3_dotlk

package adiantum_test

Expand Down
2 changes: 1 addition & 1 deletion vfs/shm.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
//go:build (darwin || linux || freebsd || openbsd || netbsd || dragonfly || illumos || sqlite3_flock) && (386 || arm || amd64 || arm64 || riscv64 || ppc64le) && !(sqlite3_dotlk || sqlite3_noshm || sqlite3_nosys)
//go:build ((darwin || linux || freebsd || openbsd || netbsd || dragonfly || illumos) && (386 || arm || amd64 || arm64 || riscv64 || ppc64le) && !sqlite3_nosys) || sqlite3_flock || sqlite3_dotlk

package vfs

Expand Down
22 changes: 10 additions & 12 deletions vfs/shm_bsd.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
//go:build (freebsd || openbsd || netbsd || dragonfly || illumos || sqlite3_flock) && (386 || arm || amd64 || arm64 || riscv64 || ppc64le) && !(sqlite3_dotlk || sqlite3_noshm || sqlite3_nosys)
//go:build ((freebsd || openbsd || netbsd || dragonfly || illumos) && (386 || arm || amd64 || arm64 || riscv64 || ppc64le) && !(sqlite3_dotlk || sqlite3_nosys)) || sqlite3_flock

package vfs

Expand All @@ -23,9 +23,9 @@ type vfsShmFile struct {
// +checklocks:vfsShmFilesMtx
refs int

// +checklocks:lockMtx
lock [_SHM_NLOCK]int16
lockMtx sync.Mutex
// +checklocks:Mutex
lock [_SHM_NLOCK]int16
sync.Mutex
}

var (
Expand Down Expand Up @@ -54,7 +54,7 @@ func (s *vfsShm) Close() error {
s.shmLock(0, _SHM_NLOCK, _SHM_UNLOCK)

// Decrease reference count.
if s.vfsShmFile.refs > 1 {
if s.vfsShmFile.refs > 0 {
s.vfsShmFile.refs--
s.vfsShmFile = nil
return nil
Expand Down Expand Up @@ -119,7 +119,6 @@ func (s *vfsShm) shmOpen() (rc _ErrorCode) {
s.vfsShmFile = &vfsShmFile{
File: f,
info: fi,
refs: 1,
}
f = nil // Don't close the file.
for i, g := range vfsShmFiles {
Expand Down Expand Up @@ -174,8 +173,8 @@ func (s *vfsShm) shmMap(ctx context.Context, mod api.Module, id, size int32, ext
}

func (s *vfsShm) shmLock(offset, n int32, flags _ShmFlag) _ErrorCode {
s.lockMtx.Lock()
defer s.lockMtx.Unlock()
s.Lock()
defer s.Unlock()

switch {
case flags&_SHM_UNLOCK != 0:
Expand Down Expand Up @@ -234,8 +233,7 @@ func (s *vfsShm) shmUnmap(delete bool) {
for _, r := range s.regions {
r.Unmap()
}
clear(s.regions)
s.regions = s.regions[:0]
s.regions = nil

// Close the file.
if delete {
Expand All @@ -245,7 +243,7 @@ func (s *vfsShm) shmUnmap(delete bool) {
}

func (s *vfsShm) shmBarrier() {
s.lockMtx.Lock()
s.Lock()
//lint:ignore SA2001 memory barrier.
s.lockMtx.Unlock()
s.Unlock()
}
Loading