Revision 616238643765 () - Diff

Link to this snippet: https://friendpaste.com/7QRfJfkG2r9w7roPLyRdMJ
Embed:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
root@ip-10-165-5-46:/tmp# more core.memcached.6895.log 
Basic crash dump analysis of core.memcached.6895.

Please send the file to support@couchbase.com

--------------------------------------------------------------------------------
File information:
-rwxr-xr-x 1 couchbase couchbase 1622012 Feb 28 20:46 /opt/couchbase/bin/memcached
16529fa2fa898005bbcb4e1255ea53a2  /opt/couchbase/bin/memcached
-rw------- 1 couchbase couchbase 4250636288 Mar  7 15:26 core.memcached.6895
58eeb7aee2b08d5bac38b7f2d6900cb1  core.memcached.6895
--------------------------------------------------------------------------------
Core file callstacks:
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from /opt/couchbase/bin/memcached...done.
[New LWP 6917]
[New LWP 8591]
[New LWP 8592]
[New LWP 8593]
[New LWP 8590]
[New LWP 6895]
[New LWP 6918]
[New LWP 6919]
[New LWP 6909]
[New LWP 6906]
[New LWP 6916]
[New LWP 6914]
[New LWP 6911]
[New LWP 6915]
[New LWP 6908]
[New LWP 6910]
[New LWP 8589]
[New LWP 6905]
[New LWP 6912]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
Program terminated with signal 11, Segmentation fault.
#0  add_conn_to_pending_io_list (c=<optimized out>) at daemon/thread.c:722
722	daemon/thread.c: No such file or directory.

Thread 19 (Thread 0x7f4bb2d94700 (LWP 6912)):
#0  0x00007f4bb6bd0353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f4bb7a30f36 in epoll_dispatch (base=0x629c780, tv=<optimized out>) at epoll.c:404
#2  0x00007f4bb7a1c394 in event_base_loop (base=0x629c780, flags=<optimized out>) at event.c:1558
#3  0x0000000000414b94 in worker_libevent (arg=0x1a377e0) at daemon/thread.c:301
#4  0x00007f4bb6ea2e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5  0x00007f4bb6bcfcbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 18 (Thread 0x7f4bb63c2700 (LWP 6905)):
#0  0x00007f4bb6bc28bd in read () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f4bb6b56ff8 in _IO_file_underflow () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f4bb6b5803e in _IO_default_uflow () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007f4bb6b4c18a in _IO_getline_info () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x00007f4bb6b4b06b in fgets () from /lib/x86_64-linux-gnu/libc.so.6
#5  0x00007f4bb63c3b19 in fgets (__stream=<optimized out>, __n=<optimized out>, __s=<optimized out>) at /usr/include/bits/stdio2.h:255
#6  check_stdin_thread (arg=<optimized out>) at extensions/daemon/stdin_check.c:37
#7  0x00007f4bb6ea2e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#8  0x00007f4bb6bcfcbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#9  0x0000000000000000 in ?? ()

Thread 17 (Thread 0x7f4baec6e700 (LWP 8589)):
#0  0x00007f4bb6eaa653 in pread64 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f4bb2093288 in pread (__offset=<optimized out>, __nbytes=<optimized out>, __buf=<optimized out>, __fd=<optimized out>) at /usr/include/bits/unistd.h:100
#2  couch_pread (handle=<optimized out>, buf=0x52084c31, nbyte=8192, offset=4112384) at src/os.c:38
#3  0x00007f4bb2345f6d in cfs_pread (h=0x28e41ce0, buf=0x52084c31, sz=8192, off=4112384) at src/couch-kvstore/couch-fs-stats.cc:68
#4  0x00007f4bb2091b1c in load_buffer_from (nbyte=<optimized out>, offset=<optimized out>, buf=<optimized out>) at src/iobuffer.c:153
#5  buffered_pread (handle=0x5de0fe60, buf=<optimized out>, nbyte=8, offset=4115426) at src/iobuffer.c:292
#6  0x00007f4bb208ffcc in read_skipping_prefixes (dst=0x7f4baec6ced0, len=<optimized out>, pos=<optimized out>, db=<optimized out>) at src/couch_file_read.c:27
#7  pread_bin_internal (db=0x51a908c0, pos=4115426, ret_ptr=<optimized out>, header=0) at src/couch_file_read.c:53
#8  0x00007f4bb20901d7 in pread_compressed (db=0x5c, pos=1376275505, ret_ptr=0x7f4baec6cfc8) at src/couch_file_read.c:92
#9  0x00007f4bb208bc9a in modify_node (rq=0x7f4baec6d260, nptr=<optimized out>, start=2, end=-1, dst=0x102000) at src/btree_modify.c:317
#10 0x00007f4bb208be8b in modify_node (rq=0x7f4baec6d260, nptr=<optimized out>, start=2, end=3, dst=<optimized out>) at src/btree_modify.c:458
#11 0x00007f4bb208be8b in modify_node (rq=0x7f4baec6d260, nptr=<optimized out>, start=1, end=190, dst=<optimized out>) at src/btree_modify.c:458
#12 0x00007f4bb208c667 in modify_btree (rq=0x7f4baec6d260, root=0x28a3cc40, errcode=0x7f4baec6d30c) at src/btree_modify.c:607
#13 0x00007f4bb208f58f in update_indexes (db=<optimized out>, seqs=0x3a268010, seqvals=0x3a268d10, ids=<optimized out>, idvals=<optimized out>, numdocs=104) at src/couch_save.c:228
#14 0x00007f4bb208fcfd in couchstore_save_documents (db=<optimized out>, docs=0xf50d4a80, infos=0x49826700, numdocs=<optimized out>, options=<optimized out>) at src/couch_save.c:359
#15 0x00007f4bb233a399 in CouchKVStore::saveDocs (this=0x6e70f00, vbid=96, rev=<optimized out>, docs=<optimized out>, docinfos=0x49826700, docCount=<optimized out>)
    at src/couch-kvstore/couch-kvstore.cc:1557
#16 0x00007f4bb233aba1 in CouchKVStore::commit2couchstore (this=0x6e70f00) at src/couch-kvstore/couch-kvstore.cc:1492
#17 0x00007f4bb233aeb9 in queueItem (req=0xf37fafc0, this=<optimized out>) at src/couch-kvstore/couch-kvstore.cc:1625
#18 CouchKVStore::set (this=0x6e70f00, itm=..., cb=...) at src/couch-kvstore/couch-kvstore.cc:344
#19 0x00007f4bb22c8c27 in EventuallyPersistentStore::flushOneDelOrSet (this=0x71c3c000, qi=..., rejectQueue=..., vb=...) at src/ep.cc:2420
#20 0x00007f4bb22d0f65 in EventuallyPersistentStore::flushOne (this=0x71c3c000, queue=..., rejectQueue=..., vb=...) at src/ep.cc:2468
#21 0x00007f4bb22d14a5 in EventuallyPersistentStore::flushVBQueue (this=0x71c3c000, vb=..., vb_queue=..., vbid=<optimized out>, data_age=<optimized out>) at src/ep.cc:2022
#22 0x00007f4bb22d26dc in EventuallyPersistentStore::flushOutgoingQueue (this=0x71c3c000, flushQueue=0x71c3c2c8, flushPhase=<optimized out>, nextVbid=<optimized out>) at src/ep.cc:1964
#23 0x00007f4bb22f9e7c in Flusher::doFlush (this=0x6307b00) at src/flusher.cc:245
#24 0x00007f4bb22fad05 in Flusher::step (this=0x5c, d=..., tid=...) at src/flusher.cc:158
#25 0x00007f4bb22c04da in Dispatcher::run (this=0x72641500) at src/dispatcher.cc:173
#26 0x00007f4bb22c0ddb in launch_dispatcher_thread (arg=0x5c) at src/dispatcher.cc:28
#27 0x00007f4bb6ea2e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#28 0x00007f4bb6bcfcbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#29 0x0000000000000000 in ?? ()

Thread 16 (Thread 0x7f4bb3d96700 (LWP 6910)):
#0  0x00007f4bb6bd0353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f4bb7a30f36 in epoll_dispatch (base=0x629cc80, tv=<optimized out>) at epoll.c:404
#2  0x00007f4bb7a1c394 in event_base_loop (base=0x629cc80, flags=<optimized out>) at event.c:1558
#3  0x0000000000414b94 in worker_libevent (arg=0x1a375f0) at daemon/thread.c:301
#4  0x00007f4bb6ea2e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5  0x00007f4bb6bcfcbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 15 (Thread 0x7f4bb4d98700 (LWP 6908)):
#0  0x00007f4bb6bd0353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f4bb7a30f36 in epoll_dispatch (base=0x629c500, tv=<optimized out>) at epoll.c:404
#2  0x00007f4bb7a1c394 in event_base_loop (base=0x629c500, flags=<optimized out>) at event.c:1558
#3  0x0000000000414b94 in worker_libevent (arg=0x1a37400) at daemon/thread.c:301
#4  0x00007f4bb6ea2e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5  0x00007f4bb6bcfcbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 14 (Thread 0x7f4bb1473700 (LWP 6915)):
#0  0x00007f4bb6eaa653 in pread64 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f4bb2093288 in pread (__offset=<optimized out>, __nbytes=<optimized out>, __buf=<optimized out>, __fd=<optimized out>) at /usr/include/bits/unistd.h:100
#2  couch_pread (handle=<optimized out>, buf=0xf377c831, nbyte=8192, offset=4579328) at src/os.c:38
#3  0x00007f4bb2345f6d in cfs_pread (h=0x2866a040, buf=0xf377c831, sz=8192, off=4579328) at src/couch-kvstore/couch-fs-stats.cc:68
#4  0x00007f4bb2091b1c in load_buffer_from (nbyte=<optimized out>, offset=<optimized out>, buf=<optimized out>) at src/iobuffer.c:153
#5  buffered_pread (handle=0xf4061920, buf=<optimized out>, nbyte=8, offset=4584790) at src/iobuffer.c:292
#6  0x00007f4bb208ffcc in read_skipping_prefixes (dst=0x7f4bb1471ed0, len=<optimized out>, pos=<optimized out>, db=<optimized out>) at src/couch_file_read.c:27
#7  pread_bin_internal (db=0x62a38f0, pos=4584790, ret_ptr=<optimized out>, header=0) at src/couch_file_read.c:53
#8  0x00007f4bb20901d7 in pread_compressed (db=0x36, pos=4084713521, ret_ptr=0x7f4bb1471fc8) at src/couch_file_read.c:92
#9  0x00007f4bb208bc9a in modify_node (rq=0x7f4bb1472200, nptr=<optimized out>, start=0, end=-1, dst=0x8000) at src/btree_modify.c:317
#10 0x00007f4bb208be8b in modify_node (rq=0x7f4bb1472200, nptr=<optimized out>, start=0, end=146, dst=<optimized out>) at src/btree_modify.c:458
#11 0x00007f4bb208be8b in modify_node (rq=0x7f4bb1472200, nptr=<optimized out>, start=0, end=152, dst=<optimized out>) at src/btree_modify.c:458
#12 0x00007f4bb208c667 in modify_btree (rq=0x7f4bb1472200, root=0x28a3df40, errcode=0x7f4bb147230c) at src/btree_modify.c:607
#13 0x00007f4bb208f465 in update_indexes (db=<optimized out>, seqs=0x3a378010, seqvals=0x3a378990, ids=<optimized out>, idvals=0x3a378e50, numdocs=76) at src/couch_save.c:204
#14 0x00007f4bb208fcfd in couchstore_save_documents (db=<optimized out>, docs=0xf3626c80, infos=0xf3626a00, numdocs=<optimized out>, options=<optimized out>) at src/couch_save.c:359
#15 0x00007f4bb233a399 in CouchKVStore::saveDocs (this=0x6308000, vbid=126, rev=<optimized out>, docs=<optimized out>, docinfos=0xf3626a00, docCount=<optimized out>)
    at src/couch-kvstore/couch-kvstore.cc:1557
#16 0x00007f4bb233aba1 in CouchKVStore::commit2couchstore (this=0x6308000) at src/couch-kvstore/couch-kvstore.cc:1492
#17 0x00007f4bb233b06c in queueItem (req=0xf37fb440, this=<optimized out>) at src/couch-kvstore/couch-kvstore.cc:1625
#18 CouchKVStore::del (this=0x6308000, itm=..., cb=...) at src/couch-kvstore/couch-kvstore.cc:510
#19 0x00007f4bb22c8ac1 in EventuallyPersistentStore::flushOneDelOrSet (this=0x62ee480, qi=..., rejectQueue=..., vb=...) at src/ep.cc:2446
#20 0x00007f4bb22d0f65 in EventuallyPersistentStore::flushOne (this=0x62ee480, queue=..., rejectQueue=..., vb=...) at src/ep.cc:2468
#21 0x00007f4bb22d14a5 in EventuallyPersistentStore::flushVBQueue (this=0x62ee480, vb=..., vb_queue=..., vbid=<optimized out>, data_age=<optimized out>) at src/ep.cc:2022
#22 0x00007f4bb22d26dc in EventuallyPersistentStore::flushOutgoingQueue (this=0x62ee480, flushQueue=0x62ee748, flushPhase=<optimized out>, nextVbid=<optimized out>) at src/ep.cc:1964
#23 0x00007f4bb22f9e7c in Flusher::doFlush (this=0x63065a0) at src/flusher.cc:245
#24 0x00007f4bb22fad05 in Flusher::step (this=0x36, d=..., tid=...) at src/flusher.cc:158
#25 0x00007f4bb22c04da in Dispatcher::run (this=0x62e8a80) at src/dispatcher.cc:173
#26 0x00007f4bb22c0ddb in launch_dispatcher_thread (arg=0x36) at src/dispatcher.cc:28
#27 0x00007f4bb6ea2e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#28 0x00007f4bb6bcfcbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#29 0x0000000000000000 in ?? ()

Thread 13 (Thread 0x7f4bb3595700 (LWP 6911)):
#0  0x00007f4bb6bd0353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f4bb7a30f36 in epoll_dispatch (base=0x629ca00, tv=<optimized out>) at epoll.c:404
#2  0x00007f4bb7a1c394 in event_base_loop (base=0x629ca00, flags=<optimized out>) at event.c:1558
#3  0x0000000000414b94 in worker_libevent (arg=0x1a376e8) at daemon/thread.c:301
#4  0x00007f4bb6ea2e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5  0x00007f4bb6bcfcbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 12 (Thread 0x7f4bb1e81700 (LWP 6914)):
#0  0x00007f4bb6b9b83d in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f4bb6bc9774 in usleep () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f4bb2300055 in updateStatsThread (arg=<optimized out>) at src/memory_tracker.cc:31
#3  0x00007f4bb6ea2e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4  0x00007f4bb6bcfcbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5  0x0000000000000000 in ?? ()

Thread 11 (Thread 0x7f4bb0c72700 (LWP 6916)):
#0  0x00007f4bb6ea70fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f4bb22bd528 in wait (tv=..., this=<optimized out>) at src/syncobject.hh:58
#2  IdleTask::run (this=0x1a3c120, d=...) at src/dispatcher.cc:336
#3  0x00007f4bb22c04da in Dispatcher::run (this=0x62e9880) at src/dispatcher.cc:173
#4  0x00007f4bb22c0ddb in launch_dispatcher_thread (arg=0x62e98d4) at src/dispatcher.cc:28
#5  0x00007f4bb6ea2e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6  0x00007f4bb6bcfcbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7  0x0000000000000000 in ?? ()

Thread 10 (Thread 0x7f4bb57a6700 (LWP 6906)):
#0  0x00007f4bb6ea70fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f4bb59c0176 in logger_thead_main (arg=<optimized out>) at extensions/loggers/file_logger.c:368
#2  0x00007f4bb6ea2e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#3  0x00007f4bb6bcfcbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x0000000000000000 in ?? ()

Thread 9 (Thread 0x7f4bb4597700 (LWP 6909)):
#0  0x00007f4bb6bd0353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f4bb7a30f36 in epoll_dispatch (base=0x629c280, tv=<optimized out>) at epoll.c:404
#2  0x00007f4bb7a1c394 in event_base_loop (base=0x629c280, flags=<optimized out>) at event.c:1558
#3  0x0000000000414b94 in worker_libevent (arg=0x1a374f8) at daemon/thread.c:301
#4  0x00007f4bb6ea2e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#5  0x00007f4bb6bcfcbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7f4baf46f700 (LWP 6919)):
#0  0x00007f4bb6ea70fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f4bb22de7ef in wait (tv=..., this=<optimized out>) at src/syncobject.hh:58
#2  wait (secs=<optimized out>, this=<optimized out>) at src/syncobject.hh:74
#3  wait (previousCounter=<optimized out>, howlong=<optimized out>, this=<optimized out>) at src/tapconnmap.hh:169
#4  EventuallyPersistentEngine::notifyPendingConnections (this=0x62f2000) at src/ep_engine.cc:3423
#5  0x00007f4bb22de8d3 in EvpNotifyPendingConns (arg=0x62f2000) at src/ep_engine.cc:1145
#6  0x00007f4bb6ea2e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7  0x00007f4bb6bcfcbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#8  0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7f4bafc70700 (LWP 6918)):
#0  0x00007f4bb6ea70fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f4bb22bd528 in wait (tv=..., this=<optimized out>) at src/syncobject.hh:58
#2  IdleTask::run (this=0x1a3c2d0, d=...) at src/dispatcher.cc:336
#3  0x00007f4bb22c04da in Dispatcher::run (this=0x62e9500) at src/dispatcher.cc:173
#4  0x00007f4bb22c0ddb in launch_dispatcher_thread (arg=0x62e9554) at src/dispatcher.cc:28
#5  0x00007f4bb6ea2e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6  0x00007f4bb6bcfcbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7  0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7f4bb806e740 (LWP 6895)):
#0  0x00007f4bb6bd0353 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f4bb7a30f36 in epoll_dispatch (base=0x629c000, tv=<optimized out>) at epoll.c:404
#2  0x00007f4bb7a1c394 in event_base_loop (base=0x629c000, flags=<optimized out>) at event.c:1558
#3  0x000000000040c2e1 in main (argc=<optimized out>, argv=<optimized out>) at daemon/memcached.c:7918

Thread 5 (Thread 0x7f4bae46d700 (LWP 8590)):
#0  0x00007f4bb6ea70fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f4bb22bd528 in wait (tv=..., this=<optimized out>) at src/syncobject.hh:58
#2  IdleTask::run (this=0x6e204c60, d=...) at src/dispatcher.cc:336
#3  0x00007f4bb22c04da in Dispatcher::run (this=0x72641340) at src/dispatcher.cc:173
#4  0x00007f4bb22c0ddb in launch_dispatcher_thread (arg=0x72641394) at src/dispatcher.cc:28
#5  0x00007f4bb6ea2e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6  0x00007f4bb6bcfcbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7  0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7f4bacc6a700 (LWP 8593)):
#0  0x00007f4bb6ea70fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f4bb22de7ef in wait (tv=..., this=<optimized out>) at src/syncobject.hh:58
#2  wait (secs=<optimized out>, this=<optimized out>) at src/syncobject.hh:74
#3  wait (previousCounter=<optimized out>, howlong=<optimized out>, this=<optimized out>) at src/tapconnmap.hh:169
#4  EventuallyPersistentEngine::notifyPendingConnections (this=0x62f3b00) at src/ep_engine.cc:3423
#5  0x00007f4bb22de8d3 in EvpNotifyPendingConns (arg=0x62f3b00) at src/ep_engine.cc:1145
#6  0x00007f4bb6ea2e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#7  0x00007f4bb6bcfcbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#8  0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7f4bad46b700 (LWP 8592)):
#0  0x00007f4bb6ea70fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f4bb22bd528 in wait (tv=..., this=<optimized out>) at src/syncobject.hh:58
#2  IdleTask::run (this=0x6ec04e10, d=...) at src/dispatcher.cc:336
#3  0x00007f4bb22c04da in Dispatcher::run (this=0x72640fc0) at src/dispatcher.cc:173
#4  0x00007f4bb22c0ddb in launch_dispatcher_thread (arg=0x72641014) at src/dispatcher.cc:28
#5  0x00007f4bb6ea2e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6  0x00007f4bb6bcfcbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7  0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7f4badc6c700 (LWP 8591)):
#0  0x00007f4bb6ea70fe in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007f4bb22bd528 in wait (tv=..., this=<optimized out>) at src/syncobject.hh:58
#2  IdleTask::run (this=0x6ec05440, d=...) at src/dispatcher.cc:336
#3  0x00007f4bb22c04da in Dispatcher::run (this=0x72641180) at src/dispatcher.cc:173
#4  0x00007f4bb22c0ddb in launch_dispatcher_thread (arg=0x726411d4) at src/dispatcher.cc:28
#5  0x00007f4bb6ea2e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6  0x00007f4bb6bcfcbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7  0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7f4bb0471700 (LWP 6917)):
#0  add_conn_to_pending_io_list (c=<optimized out>) at daemon/thread.c:722
#1  notify_io_complete (cookie=0x61adb80, status=ENGINE_SUCCESS) at daemon/thread.c:488
#2  0x00007f4bb231ab8d in notifyIOComplete (status=<optimized out>, cookie=<optimized out>, this=<optimized out>) at src/ep_engine.h:439
#3  TapConnMap::notifyPausedConnection_UNLOCKED (this=<optimized out>, tc=0x71b6c00) at src/tapconnmap.cc:347
#4  0x00007f4bb231679b in TapConnMap::performTapOp<Item*> (this=0x6306000, name=..., tapop=..., arg=0x28a2fc00) at src/tapconnmap.hh:119
#5  0x00007f4bb2318d89 in TapBGFetchCallback::callback(Dispatcher&, SingleThreadedRCPtr<Task>&) () from /opt/couchbase/lib/memcached/ep.so
#6  0x00007f4bb22c04da in Dispatcher::run (this=0x62e96c0) at src/dispatcher.cc:173
#7  0x00007f4bb22c0ddb in launch_dispatcher_thread (arg=0x1a37798) at src/dispatcher.cc:28
#8  0x00007f4bb6ea2e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#9  0x00007f4bb6bcfcbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x0000000000000000 in ?? ()
--------------------------------------------------------------------------------
Module information: