Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

When setting concurrentWriters=0 A fatal exception occurs - SIGSEGV #163

Open
spastorclovr opened this issue Apr 8, 2024 · 0 comments
Open

When setting concurrentWriters=0 A fatal exception occurs - SIGSEGV #163

spastorclovr opened this issue Apr 8, 2024 · 0 comments

Comments

@spastorclovr
Copy link
Contributor

I have been trying to set concurrentWriters to 0 to make sure no limits are being put in the number of writers trying to write on the mounted volume.

I got it working at some point and it did improve a lot the writing thoughput for our service.
But, I noticied that the filerUI did not display the same file listing as the one I could see through the mounted volume.
So, I tried from scratch (shutting down all pods and up again) , and I have now the error below 👇
(Adjusting -cacheCapacityMB did not make any difference)

Seaweedfs version: 3.63
Seaweedfs-csi helm version: 0.2.1

I0408 08:06:59.211052 nodeserver.go:103 node publish volume pvc-6c3e3e3e-68d1-44cb-bfdd-da9ba5073c30 to /var/lib/kubelet/pods/138db1a7-7bda-48d0-844c-c4735a7220c1/volumes/kubernetes.io~csi/pvc-6c3e3e3e-68d1-44cb-bfdd-da9ba5073c30/mount
I0408 08:06:59.211383 nodeserver.go:137 volume pvc-80c9113b-5473-42ed-8041-c54a02cc28b7 successfully published to /var/lib/kubelet/pods/138db1a7-7bda-48d0-844c-c4735a7220c1/volumes/kubernetes.io~csi/pvc-80c9113b-5473-42ed-8041-c54a02cc28b7/mount
I0408 08:06:59.211409 utils.go:68 GRPC /csi.v1.Node/NodePublishVolume response
I0408 08:06:59.212475 nodeserver.go:137 volume pvc-6c3e3e3e-68d1-44cb-bfdd-da9ba5073c30 successfully published to /var/lib/kubelet/pods/138db1a7-7bda-48d0-844c-c4735a7220c1/volumes/kubernetes.io~csi/pvc-6c3e3e3e-68d1-44cb-bfdd-da9ba5073c30/mount
I0408 08:06:59.212492 utils.go:68 GRPC /csi.v1.Node/NodePublishVolume response
I0408 08:07:01.241693 meta_cache_init.go:44 ReadDirAllEntries /buckets/pvc-bc59ef4b-5919-48f4-8067-d90e0efd1624 ...
I0408 08:07:01.241748 filer_client.go:120 read directory: directory:"/buckets/pvc-bc59ef4b-5919-48f4-8067-d90e0efd1624" limit:10001
I0408 08:07:01.243415 weedfs_dir_lookup.go:51 dir Lookup cache hit /buckets/pvc-bc59ef4b-5919-48f4-8067-d90e0efd1624/bitcoin.conf
I0408 08:07:01.363138 weedfs_dir_lookup.go:51 dir Lookup cache hit /buckets/pvc-bc59ef4b-5919-48f4-8067-d90e0efd1624/settings.json
ActiveLock 1 Read locked 0 type=0 with waiters 0 active r0 w0.
I0408 08:07:01.368452 reader_at.go:179 zero2 [4,4096) of file size 4 bytes
I0408 08:07:01.368480 page_writer.go:54 ReadDirtyDataAt 0 [0, 4096)
ActiveLock 1 Read unlocked 0 type=0 with waiters 0 active r0 w0.
I0408 08:07:01.368543 weedfs_file_sync.go:98 doFlush /buckets/pvc-bc59ef4b-5919-48f4-8067-d90e0efd1624/settings.json fh 0
ActiveLock 2 doFlush wait for 0 type=1 with waiters 0 active r0 w0.
ActiveLock 2 doFlush locked 0 type=1 with waiters 0 active r0 w0.
ActiveLock 2 doFlush unlocked 0 type=1 with waiters 0 active r0 w0.
ActiveLock 3 ReleaseHandle wait for 0 type=1 with waiters 0 active r0 w0.
ActiveLock 3 ReleaseHandle locked 0 type=1 with waiters 0 active r0 w0.
ActiveLock 3 ReleaseHandle unlocked 0 type=1 with waiters 0 active r0 w0.
I0408 08:07:01.368718 weedfs_dir_lookup.go:51 dir Lookup cache hit /buckets/pvc-bc59ef4b-5919-48f4-8067-d90e0efd1624/settings.json.tmp
I0408 08:07:01.368866 weedfs_attr.go:53 /buckets/pvc-bc59ef4b-5919-48f4-8067-d90e0efd1624/settings.json.tmp setattr set size=0 chunks=0
ActiveLock 4 Write wait for 1 type=1 with waiters 0 active r0 w0.
ActiveLock 4 Write locked 1 type=1 with waiters 0 active r0 w0.
I0408 08:07:01.368977 page_writer.go:34 1 AddPage [0, 4)
I0408 08:07:01.368988 dirty_pages_chunked.go:44 1 memory AddPage [0, 4)
ActiveLock 4 Write unlocked 1 type=1 with waiters 0 active r0 w0.
I0408 08:07:01.369174 weedfs_file_sync.go:98 doFlush /buckets/pvc-bc59ef4b-5919-48f4-8067-d90e0efd1624/settings.json.tmp fh 1
I0408 08:07:01.369184 upload_pipeline.go:171  uploaderCount 0 ++> 1
fatal error: sync: unlock of unlocked mutex
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xcf2b85]

goroutine 117 [running]:
sync.fatal({0x23d8342?, 0x44?})
	/usr/local/go/src/runtime/panic.go:1061 +0x18
sync.(*Mutex).unlockSlow(0xc0005bc828, 0xffffffff)
	/usr/local/go/src/sync/mutex.go:229 +0x35
sync.(*Mutex).Unlock(0x0?)
	/usr/local/go/src/sync/mutex.go:223 +0x25
panic({0x1fbc240?, 0x3d121f0?})
	/usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/seaweedfs/seaweedfs/weed/util.(*LimitedConcurrentExecutor).Execute(0x0, 0xc00070c520)
	/go/src/github.com/seaweedfs/seaweedfs/weed/util/limited_executor.go:32 +0x25
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).moveToSealed(0xc0005bc7e0, {0x2a57d90?, 0xc0006cfd80}, 0x0)
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/page_writer/upload_pipeline.go:185 +0x325
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).flushChunks(0xc0005bc7e0)
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/page_writer/upload_pipeline.go:159 +0xe5
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).FlushAll(0x0?)
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/page_writer/upload_pipeline.go:150 +0x18
github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).FlushData(0xc0006d16e0)
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/dirty_pages_chunked.go:54 +0x29
github.com/seaweedfs/seaweedfs/weed/mount.(*PageWriter).FlushData(...)
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/page_writer.go:50
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).doFlush(0xc0006ec840, 0xc0005fa180, 0x3e8, 0x3e8)
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/weedfs_file_sync.go:101 +0x169
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).Flush(0xc0006ec840, 0x7f31eebc1f18?, 0xc0007123d8)
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/weedfs_file_sync.go:59 +0x48
github.com/hanwen/go-fuse/v2/fuse.doFlush(0xc000712240?, 0xc000712240)
	/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.4.0/fuse/opcode.go:402 +0x2c
github.com/hanwen/go-fuse/v2/fuse.(*Server).handleRequest(0xc0002d0f20, 0xc000712240)
	/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.4.0/fuse/server.go:526 +0x267
github.com/hanwen/go-fuse/v2/fuse.(*Server).loop(0xc0002d0f20, 0x0?)
	/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.4.0/fuse/server.go:499 +0x110
created by github.com/hanwen/go-fuse/v2/fuse.(*Server).readRequest in goroutine 1
	/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.4.0/fuse/server.go:366 +0x53f

goroutine 1 [syscall]:
syscall.Syscall(0x4800a5?, 0xc0002d0fb8?, 0x47ffea?, 0x1e6ea60?)
	/usr/local/go/src/syscall/syscall_linux.go:69 +0x25
syscall.read(0xc0002d0fb8?, {0xc0019881b0?, 0xc0007e29f0?, 0xc00095d828?})
	/usr/local/go/src/syscall/zsyscall_linux_amd64.go:721 +0x38
syscall.Read(...)
	/usr/local/go/src/syscall/syscall_unix.go:181
github.com/hanwen/go-fuse/v2/fuse.(*Server).readRequest.func1(...)
	/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.4.0/fuse/server.go:334
github.com/hanwen/go-fuse/v2/fuse.handleEINTR(...)
	/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.4.0/fuse/server.go:309
github.com/hanwen/go-fuse/v2/fuse.(*Server).readRequest(0xc0002d0f20, 0xc0?)
	/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.4.0/fuse/server.go:332 +0x1ac
github.com/hanwen/go-fuse/v2/fuse.(*Server).loop(0xc0002d0f20, 0x38?)
	/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.4.0/fuse/server.go:477 +0x7b
github.com/hanwen/go-fuse/v2/fuse.(*Server).Serve(0xc0002d0f20)
	/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.4.0/fuse/server.go:421 +0x29
github.com/seaweedfs/seaweedfs/weed/command.RunMount(0x3d4c4e0, 0x0)
	/go/src/github.com/seaweedfs/seaweedfs/weed/command/mount_std.go:271 +0x1712
github.com/seaweedfs/seaweedfs/weed/command.runMount(0x3d2fd78?, {0xc000138260?, 0x0, 0xa?})
	/go/src/github.com/seaweedfs/seaweedfs/weed/command/mount_std.go:55 +0x154
main.main()
	/go/src/github.com/seaweedfs/seaweedfs/weed/weed.go:80 +0x309

goroutine 19 [chan receive]:
github.com/seaweedfs/seaweedfs/weed/glog.(*loggingT).flushDaemon(0x0?)
	/go/src/github.com/seaweedfs/seaweedfs/weed/glog/glog.go:884 +0x66
created by github.com/seaweedfs/seaweedfs/weed/glog.init.0 in goroutine 1
	/go/src/github.com/seaweedfs/seaweedfs/weed/glog/glog.go:409 +0x1ae

goroutine 24 [syscall]:
os/signal.signal_recv()
	/usr/local/go/src/runtime/sigqueue.go:152 +0x29
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x13
created by os/signal.Notify.func1.1 in goroutine 1
	/usr/local/go/src/os/signal/signal.go:151 +0x1f

goroutine 50 [chan receive]:
github.com/seaweedfs/seaweedfs/weed/util/grace.init.0.func1()
	/go/src/github.com/seaweedfs/seaweedfs/weed/util/grace/signal_handling.go:38 +0x45
created by github.com/seaweedfs/seaweedfs/weed/util/grace.init.0 in goroutine 1
	/go/src/github.com/seaweedfs/seaweedfs/weed/util/grace/signal_handling.go:37 +0x113

goroutine 51 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0001cab00)
	/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f
created by go.opencensus.io/stats/view.init.0 in goroutine 1
	/go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d

goroutine 82 [select]:
github.com/seaweedfs/seaweedfs/weed/stats.(*ServerStats).Start(0xc0000614c0)
	/go/src/github.com/seaweedfs/seaweedfs/weed/stats/stats.go:92 +0x1c5
created by github.com/seaweedfs/seaweedfs/weed/server.init.0 in goroutine 1
	/go/src/github.com/seaweedfs/seaweedfs/weed/server/common.go:41 +0x7b

goroutine 83 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0002cd9c0, {0x2a4d518, 0xc0007cde00})
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:76 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:52 +0x129

goroutine 84 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0002cd9f0, {0x2a4d518, 0xc0007cde50})
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:76 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:52 +0x129

goroutine 85 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0002cda20, {0x2a4d518, 0xc0007cdea0})
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:76 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:52 +0x129

goroutine 42 [IO wait]:
internal/poll.runtime_pollWait(0x7f31a7fa8dc0, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc0007c2000?, 0xc000282000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0007c2000, {0xc000282000, 0x8000, 0x8000})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc0007c2000, {0xc000282000?, 0x1060100000000?, 0x8?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc0005be0f0, {0xc000282000?, 0x18?, 0xc0004c3c80?})
	/usr/local/go/src/net/net.go:179 +0x45
bufio.(*Reader).Read(0xc0000b6660, {0xc0006b8120, 0x9, 0x7f31eebc1f18?})
	/usr/local/go/src/bufio/bufio.go:244 +0x197
io.ReadAtLeast({0x2a2dfe0, 0xc0000b6660}, {0xc0006b8120, 0x9, 0x9}, 0x9)
	/usr/local/go/src/io/io.go:335 +0x90
io.ReadFull(...)
	/usr/local/go/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc0006b8120, 0x9, 0x3d4b520?}, {0x2a2dfe0?, 0xc0000b6660?})
	/go/pkg/mod/golang.org/x/net@v0.20.0/http2/frame.go:237 +0x65
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0006b80e0)
	/go/pkg/mod/golang.org/x/net@v0.20.0/http2/frame.go:498 +0x85
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0007e4480, 0x0?)
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:1587 +0x245
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 86
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:398 +0x1ebe

goroutine 41 [select]:
google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc0007e4480)
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:1661 +0x151
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 86
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:388 +0x1e4b

goroutine 43 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0006e6370, 0x1)
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/controlbuf.go:418 +0x113
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0000361c0)
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/controlbuf.go:552 +0x86
google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:452 +0x85
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 86
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:450 +0x246b

goroutine 101 [select]:
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc00092e1c0)
	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190318030020-c3a204f8e965/leveldb/util/buffer_pool.go:206 +0xb6
created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool in goroutine 1
	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190318030020-c3a204f8e965/leveldb/util/buffer_pool.go:237 +0x196

goroutine 102 [select]:
github.com/syndtr/goleveldb/leveldb.(*session).refLoop(0xc00017c780)
	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190318030020-c3a204f8e965/leveldb/session_util.go:189 +0x59e
created by github.com/syndtr/goleveldb/leveldb.newSession in goroutine 1
	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190318030020-c3a204f8e965/leveldb/session.go:93 +0x2a6

goroutine 103 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc0002dfba0)
	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190318030020-c3a204f8e965/leveldb/db_compaction.go:90 +0x12a
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 1
	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190318030020-c3a204f8e965/leveldb/db.go:142 +0x456

goroutine 104 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc0002dfba0)
	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190318030020-c3a204f8e965/leveldb/db_state.go:101 +0x9c
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 1
	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190318030020-c3a204f8e965/leveldb/db.go:143 +0x496

goroutine 105 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc0002dfba0)
	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190318030020-c3a204f8e965/leveldb/db_compaction.go:825 +0x698
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 1
	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190318030020-c3a204f8e965/leveldb/db.go:149 +0x505

goroutine 106 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc0002dfba0)
	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190318030020-c3a204f8e965/leveldb/db_compaction.go:762 +0x105
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 1
	/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20190318030020-c3a204f8e965/leveldb/db.go:150 +0x545

goroutine 114 [IO wait]:
internal/poll.runtime_pollWait(0x7f31a7fa8eb8, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc0001cb600?, 0x20?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0001cb600)
	/usr/local/go/src/internal/poll/fd_unix.go:611 +0x2ac
net.(*netFD).accept(0xc0001cb600)
	/usr/local/go/src/net/fd_unix.go:172 +0x29
net.(*UnixListener).accept(0x448e60?)
	/usr/local/go/src/net/unixsock_posix.go:172 +0x16
net.(*UnixListener).Accept(0xc000982d50)
	/usr/local/go/src/net/unixsock.go:260 +0x30
google.golang.org/grpc.(*Server).Serve(0xc0004d0800, {0x2a465e0?, 0xc000982d50})
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/server.go:871 +0x462
created by github.com/seaweedfs/seaweedfs/weed/command.RunMount in goroutine 1
	/go/src/github.com/seaweedfs/seaweedfs/weed/command/mount_std.go:264 +0x1545

goroutine 115 [select]:
google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc0007de120)
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/transport.go:329 +0x7c
google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/transport.go:344
google.golang.org/grpc.(*csAttempt).recvMsg(0xc0006cb040, {0x21c50a0?, 0xc0005c84b0}, 0x41aae8?)
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/stream.go:1070 +0xc9
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x52?)
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/stream.go:927 +0x1f
google.golang.org/grpc.(*clientStream).withRetry(0xc000706fc0, 0xc000763850, 0xc000763840)
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/stream.go:776 +0x13a
google.golang.org/grpc.(*clientStream).RecvMsg(0xc000706fc0, {0x21c50a0?, 0xc0005c84b0?})
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/stream.go:926 +0x125
github.com/seaweedfs/seaweedfs/weed/pb/filer_pb.(*seaweedFilerSubscribeMetadataClient).Recv(0xc0006765c0)
	/go/src/github.com/seaweedfs/seaweedfs/weed/pb/filer_pb/filer_grpc.pb.go:294 +0x46
github.com/seaweedfs/seaweedfs/weed/pb.WithFilerClientFollowMetadata.makeSubscribeMetadataFunc.func1({0x2a699b8, 0xc000984ad0})
	/go/src/github.com/seaweedfs/seaweedfs/weed/pb/filer_pb_tail.go:78 +0x23f
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).WithFilerClient.func1.1(0xc00065c380)
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/wfs_filer_client.go:27 +0x68
github.com/seaweedfs/seaweedfs/weed/pb.WithGrpcClient(0xaf?, 0x5bfc52d2?, 0xc000763ce8, {0xc000129290, 0x26}, 0x0?, {0xc000094cd8?, 0x1?, 0x1?})
	/go/src/github.com/seaweedfs/seaweedfs/weed/pb/grpc_client_server.go:155 +0x348
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).WithFilerClient.func1()
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/wfs_filer_client.go:25 +0x15e
github.com/seaweedfs/seaweedfs/weed/util.Retry({0x23a1371, 0xa}, 0xc000763df8)
	/go/src/github.com/seaweedfs/seaweedfs/weed/util/retry.go:16 +0xa7
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).WithFilerClient(0x0?, 0x0?, 0x0?)
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/wfs_filer_client.go:18 +0x65
github.com/seaweedfs/seaweedfs/weed/pb.WithFilerClientFollowMetadata({0x2a45f20, 0xc0006ec840}, 0xc0007c3980, 0xc0007e2660)
	/go/src/github.com/seaweedfs/seaweedfs/weed/pb/filer_pb_tail.go:50 +0x87
github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.SubscribeMetaEvents.func2()
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/meta_cache/meta_cache_subscribe.go:74 +0x26
github.com/seaweedfs/seaweedfs/weed/util.RetryUntil({0x23b086f, 0x11}, 0xc000763f60, 0x24e8368)
	/go/src/github.com/seaweedfs/seaweedfs/weed/util/retry.go:64 +0x4e
github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.SubscribeMetaEvents(0xc0007fafc0, 0x5bfc52d2, {0x2a45f20?, 0xc0006ec840}, {0x7ffcab9b563e, 0x31}, 0x17c43f89c0ca0d72)
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/meta_cache/meta_cache_subscribe.go:72 +0x185
created by github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).StartBackgroundTasks in goroutine 1
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/weedfs.go:123 +0x152

goroutine 116 [sleep]:
time.Sleep(0xe33e22200)
	/usr/local/go/src/runtime/time.go:195 +0x125
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).loopCheckQuota(0xc0006ec840)
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/weedfs_quota.go:15 +0x31
created by github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).StartBackgroundTasks in goroutine 1
	/go/src/github.com/seaweedfs/seaweedfs/weed/mount/weedfs.go:124 +0x18e

goroutine 70 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0009847b0, {0x2a4d518, 0xc0000c13b0})
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:76 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 115
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:52 +0x129

goroutine 71 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0009847e0, {0x2a4d518, 0xc0000c1450})
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:76 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 115
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:52 +0x129

goroutine 72 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000984810, {0x2a4d518, 0xc0000c14a0})
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:76 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 115
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:52 +0x129

goroutine 91 [select]:
google.golang.org/grpc.newClientStreamWithParams.func4()
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/stream.go:391 +0x8c
created by google.golang.org/grpc.newClientStreamWithParams in goroutine 115
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/stream.go:390 +0xe9c

goroutine 88 [select]:
google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc0004d66c0)
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:1661 +0x151
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 73
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:388 +0x1e4b

goroutine 89 [IO wait]:
internal/poll.runtime_pollWait(0x7f31a7fa8cc8, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc0005ba000?, 0xc000786000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0005ba000, {0xc000786000, 0x8000, 0x8000})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc0005ba000, {0xc000786000?, 0x60100000000?, 0x8?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc000782000, {0xc000786000?, 0x40b3fe?, 0x4726b2?})
	/usr/local/go/src/net/net.go:179 +0x45
bufio.(*Reader).Read(0xc0007faba0, {0xc00092e040, 0x9, 0xc0?})
	/usr/local/go/src/bufio/bufio.go:244 +0x197
io.ReadAtLeast({0x2a2dfe0, 0xc0007faba0}, {0xc00092e040, 0x9, 0x9}, 0x9)
	/usr/local/go/src/io/io.go:335 +0x90
io.ReadFull(...)
	/usr/local/go/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc00092e040, 0x9, 0x20b6300?}, {0x2a2dfe0?, 0xc0007faba0?})
	/go/pkg/mod/golang.org/x/net@v0.20.0/http2/frame.go:237 +0x65
golang.org/x/net/http2.(*Framer).ReadFrame(0xc00092e000)
	/go/pkg/mod/golang.org/x/net@v0.20.0/http2/frame.go:498 +0x85
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0004d66c0, 0x19320b2801200218?)
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:1587 +0x245
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 73
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:398 +0x1ebe

goroutine 90 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0002f1e50, 0x1)
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/controlbuf.go:418 +0x113
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000812000)
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/controlbuf.go:552 +0x86
google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:452 +0x85
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 73
	/go/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:450 +0x246b

goroutine 78 [IO wait]:
internal/poll.runtime_pollWait(0x7f31a7fa8bd0, 0x72)
	/usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc0005fa100?, 0xc0005fe000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0005fa100, {0xc0005fe000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc0005fa100, {0xc0005fe000?, 0x751120?, 0xc0005cc000?})
	/usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc0007c6108, {0xc0005fe000?, 0x0?, 0xc0005036c0?})
	/usr/local/go/src/net/net.go:179 +0x45
net/http.(*persistConn).Read(0xc000594480, {0xc0005fe000?, 0xc0001036e0?, 0xc000099d38?})
	/usr/local/go/src/net/http/transport.go:1954 +0x4a
bufio.(*Reader).fill(0xc0006d15c0)
	/usr/local/go/src/bufio/bufio.go:113 +0x103
bufio.(*Reader).Peek(0xc0006d15c0, 0x1)
	/usr/local/go/src/bufio/bufio.go:151 +0x53
net/http.(*persistConn).readLoop(0xc000594480)
	/usr/local/go/src/net/http/transport.go:2118 +0x1b9
created by net/http.(*Transport).dialConn in goroutine 119
	/usr/local/go/src/net/http/transport.go:1776 +0x169f

goroutine 79 [select]:
net/http.(*persistConn).writeLoop(0xc000594480)
	/usr/local/go/src/net/http/transport.go:2421 +0xe5
created by net/http.(*Transport).dialConn in goroutine 119
	/usr/local/go/src/net/http/transport.go:1777 +0x16f1
E0408 08:07:01.375783 mounter.go:75 weed mount exit, pid: 23, path: /var/lib/kubelet/plugins/kubernetes.io/csi/seaweedfs-csi-driver/3fbaadb3372904bcd46022cd9d47f5f22451cea3793cf1b8c028f2d721a0287b/globalmount, error: exit status 2
I0408 08:07:16.502410 utils.go:63 GRPC /csi.v1.Identity/Probe request
@spastorclovr spastorclovr changed the title Wen setting concurrentWriters=0 A fatal exception occurs - SIGSEGV When setting concurrentWriters=0 A fatal exception occurs - SIGSEGV Apr 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@spastorclovr