Revision 366462633534 () - Diff

Link to this snippet: https://friendpaste.com/4gUHpCTiJLdRDYYzp0hRUd
Embed:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
./testrunner -i /tmp/6-win-xdcr.ini get-cbcollect-info=True -t xdcr.biXDCR.bidirectional.load_with_failover,replicas=1,items=10000,ctopology=chain,rdirection=bidirection,doc-ops=create-update-delete,doc-ops-dest=create,update,failover=source

Test Input params:
{'case_number': 15,
 'cluster_name': '6-win-xdcr',
 'conf_file': '/tmp/six-node-xdcr.conf',
 'ctopology': 'chain',
 'doc-ops': 'create-update-delete',
 'doc-ops-dest': 'create,update',
 'failover': 'source',
 'get-cbcollect-info': 'True',
 'ini': '/tmp/6-win-xdcr.ini',
 'items': '10000',
 'num_nodes': 6,
 'rdirection': 'bidirection',
 'replicas': '1',
 'spec': 'six-node-xdcr'}
[2013-05-20 17:01:49,465] - [xdcrbasetests:204] INFO - Initializing input parameters started...
[2013-05-20 17:01:49,466] - [xdcrbasetests:301] INFO - Initializing input parameters completed.
[2013-05-20 17:01:49,468] - [xdcrbasetests:84] INFO - ==============  XDCRbasetests setup was started for test #15 load_with_failover==============
[2013-05-20 17:01:49,527] - [xdcrbasetests:408] INFO - cleanup cluster1: [ip:10.3.2.144 port:8091 ssh_username:Administrator, ip:10.3.2.244 port:8091 ssh_username:Administrator, ip:10.3.2.250 port:8091 ssh_username:Administrator]
[2013-05-20 17:01:49,544] - [bucket_helper:136] INFO - deleting existing buckets [] on 10.3.2.144
[2013-05-20 17:01:49,599] - [cluster_helper:84] INFO - waiting for ns_server @ 10.3.2.144:8091
[2013-05-20 17:01:49,611] - [cluster_helper:86] INFO - ns_server @ 10.3.2.144:8091 is running
[2013-05-20 17:01:49,626] - [bucket_helper:136] INFO - deleting existing buckets [] on 10.3.2.244
[2013-05-20 17:01:49,670] - [cluster_helper:84] INFO - waiting for ns_server @ 10.3.2.244:8091
[2013-05-20 17:01:49,679] - [cluster_helper:86] INFO - ns_server @ 10.3.2.244:8091 is running
[2013-05-20 17:01:49,692] - [bucket_helper:136] INFO - deleting existing buckets [] on 10.3.2.250
[2013-05-20 17:01:49,733] - [cluster_helper:84] INFO - waiting for ns_server @ 10.3.2.250:8091
[2013-05-20 17:01:49,741] - [cluster_helper:86] INFO - ns_server @ 10.3.2.250:8091 is running
[2013-05-20 17:01:49,742] - [xdcrbasetests:408] INFO - cleanup cluster2: [ip:10.3.2.125 port:8091 ssh_username:Administrator, ip:10.3.2.123 port:8091 ssh_username:Administrator, ip:10.3.2.124 port:8091 ssh_username:Administrator]
[2013-05-20 17:01:49,758] - [bucket_helper:136] INFO - deleting existing buckets [] on 10.3.2.125
[2013-05-20 17:01:49,804] - [cluster_helper:84] INFO - waiting for ns_server @ 10.3.2.125:8091
[2013-05-20 17:01:49,814] - [cluster_helper:86] INFO - ns_server @ 10.3.2.125:8091 is running
[2013-05-20 17:01:49,831] - [bucket_helper:136] INFO - deleting existing buckets [] on 10.3.2.123
[2013-05-20 17:01:49,867] - [cluster_helper:84] INFO - waiting for ns_server @ 10.3.2.123:8091
[2013-05-20 17:01:49,876] - [cluster_helper:86] INFO - ns_server @ 10.3.2.123:8091 is running
[2013-05-20 17:01:49,890] - [bucket_helper:136] INFO - deleting existing buckets [] on 10.3.2.124
[2013-05-20 17:01:49,928] - [cluster_helper:84] INFO - waiting for ns_server @ 10.3.2.124:8091
[2013-05-20 17:01:49,937] - [cluster_helper:86] INFO - ns_server @ 10.3.2.124:8091 is running
[2013-05-20 17:01:50,682] - [rest_client:1488] INFO - Consistent-views during rebalance was set as indexAwareRebalanceDisabled=true
[2013-05-20 17:01:50,683] - [rest_client:704] INFO - settings/web params on 10.3.2.144:8091 :username=Administrator&password=password&port=8091
[2013-05-20 17:01:50,910] - [rest_client:724] INFO - pools/default params : memoryQuota=2184&username=Administrator&password=password
[2013-05-20 17:01:51,330] - [rest_client:1488] INFO - Consistent-views during rebalance was set as indexAwareRebalanceDisabled=true
[2013-05-20 17:01:51,331] - [rest_client:704] INFO - settings/web params on 10.3.2.244:8091 :username=Administrator&password=password&port=8091
[2013-05-20 17:01:51,565] - [rest_client:724] INFO - pools/default params : memoryQuota=2184&username=Administrator&password=password
[2013-05-20 17:01:51,984] - [rest_client:1488] INFO - Consistent-views during rebalance was set as indexAwareRebalanceDisabled=true
[2013-05-20 17:01:51,984] - [rest_client:704] INFO - settings/web params on 10.3.2.250:8091 :username=Administrator&password=password&port=8091
[2013-05-20 17:01:52,214] - [rest_client:724] INFO - pools/default params : memoryQuota=2184&username=Administrator&password=password
[2013-05-20 17:01:53,422] - [task:260] INFO - adding node 10.3.2.244:8091 to cluster
[2013-05-20 17:01:53,422] - [rest_client:839] INFO - adding remote node @10.3.2.244:8091 to this cluster @10.3.2.144:8091
[2013-05-20 17:01:54,256] - [task:260] INFO - adding node 10.3.2.250:8091 to cluster
[2013-05-20 17:01:54,257] - [rest_client:839] INFO - adding remote node @10.3.2.250:8091 to this cluster @10.3.2.144:8091
[2013-05-20 17:01:55,254] - [rest_client:925] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.3.2.250%2Cns_1%4010.3.2.144%2Cns_1%4010.3.2.244
[2013-05-20 17:01:55,463] - [rest_client:929] INFO - rebalance operation started
[2013-05-20 17:01:55,486] - [rest_client:1031] INFO - rebalance percentage : 0 %
[2013-05-20 17:02:05,508] - [task:340] INFO - rebalancing was completed with progress: 100% in 10.0440838337 sec
[2013-05-20 17:02:06,521] - [rest_client:1294] INFO - http://10.3.2.144:8091/pools/default/buckets with param: proxyPort=11211&bucketType=membase&authType=sasl&replicaIndex=1&name=default&saslPassword=None&flushEnabled=1&replicaNumber=1&ramQuotaMB=1456
[2013-05-20 17:02:06,753] - [rest_client:1300] INFO - 0.231631994247 seconds to create bucket default
[2013-05-20 17:02:06,754] - [bucket_helper:292] INFO - waiting for memcached bucket : default in 10.3.2.144 to accept set ops
[2013-05-20 17:02:08,567] - [data_helper:289] INFO - creating direct client 10.3.2.144:11210 default
[2013-05-20 17:02:08,980] - [data_helper:289] INFO - creating direct client 10.3.2.244:11210 default
[2013-05-20 17:02:09,601] - [data_helper:289] INFO - creating direct client 10.3.2.250:11210 default
[2013-05-20 17:02:10,494] - [data_helper:289] INFO - creating direct client 10.3.2.144:11210 default
[2013-05-20 17:02:11,023] - [data_helper:289] INFO - creating direct client 10.3.2.244:11210 default
[2013-05-20 17:02:11,879] - [data_helper:289] INFO - creating direct client 10.3.2.250:11210 default
[2013-05-20 17:02:13,858] - [task:182] INFO - bucket 'default' was created with per node RAM quota: 1456
[2013-05-20 17:02:15,075] - [rest_client:1488] INFO - Consistent-views during rebalance was set as indexAwareRebalanceDisabled=true
[2013-05-20 17:02:15,075] - [rest_client:704] INFO - settings/web params on 10.3.2.125:8091 :username=Administrator&password=password&port=8091
[2013-05-20 17:02:15,298] - [rest_client:724] INFO - pools/default params : memoryQuota=2184&username=Administrator&password=password
[2013-05-20 17:02:15,720] - [rest_client:1488] INFO - Consistent-views during rebalance was set as indexAwareRebalanceDisabled=true
[2013-05-20 17:02:15,721] - [rest_client:704] INFO - settings/web params on 10.3.2.123:8091 :username=Administrator&password=password&port=8091
[2013-05-20 17:02:15,948] - [rest_client:724] INFO - pools/default params : memoryQuota=2184&username=Administrator&password=password
[2013-05-20 17:02:16,366] - [rest_client:1488] INFO - Consistent-views during rebalance was set as indexAwareRebalanceDisabled=true
[2013-05-20 17:02:16,366] - [rest_client:704] INFO - settings/web params on 10.3.2.124:8091 :username=Administrator&password=password&port=8091
[2013-05-20 17:02:16,585] - [rest_client:724] INFO - pools/default params : memoryQuota=2184&username=Administrator&password=password
[2013-05-20 17:02:17,790] - [task:260] INFO - adding node 10.3.2.123:8091 to cluster
[2013-05-20 17:02:17,790] - [rest_client:839] INFO - adding remote node @10.3.2.123:8091 to this cluster @10.3.2.125:8091
[2013-05-20 17:02:18,563] - [task:260] INFO - adding node 10.3.2.124:8091 to cluster
[2013-05-20 17:02:18,563] - [rest_client:839] INFO - adding remote node @10.3.2.124:8091 to this cluster @10.3.2.125:8091
[2013-05-20 17:02:19,848] - [rest_client:925] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.3.2.125%2Cns_1%4010.3.2.124%2Cns_1%4010.3.2.123
[2013-05-20 17:02:20,054] - [rest_client:929] INFO - rebalance operation started
[2013-05-20 17:02:20,063] - [rest_client:1031] INFO - rebalance percentage : 0 %
[2013-05-20 17:02:30,087] - [task:340] INFO - rebalancing was completed with progress: 100% in 10.0319011211 sec
[2013-05-20 17:02:31,099] - [rest_client:1294] INFO - http://10.3.2.125:8091/pools/default/buckets with param: proxyPort=11211&bucketType=membase&authType=sasl&replicaIndex=1&name=default&saslPassword=None&flushEnabled=1&replicaNumber=1&ramQuotaMB=1456
[2013-05-20 17:02:31,319] - [rest_client:1300] INFO - 0.219950914383 seconds to create bucket default
[2013-05-20 17:02:31,320] - [bucket_helper:292] INFO - waiting for memcached bucket : default in 10.3.2.125 to accept set ops
[2013-05-20 17:02:33,457] - [data_helper:289] INFO - creating direct client 10.3.2.123:11210 default
[2013-05-20 17:02:33,884] - [data_helper:289] INFO - creating direct client 10.3.2.124:11210 default
[2013-05-20 17:02:34,549] - [data_helper:289] INFO - creating direct client 10.3.2.125:11210 default
[2013-05-20 17:02:35,235] - [data_helper:289] INFO - creating direct client 10.3.2.123:11210 default
[2013-05-20 17:02:36,030] - [data_helper:289] INFO - creating direct client 10.3.2.124:11210 default
[2013-05-20 17:02:36,709] - [data_helper:289] INFO - creating direct client 10.3.2.125:11210 default
[2013-05-20 17:02:39,370] - [task:182] INFO - bucket 'default' was created with per node RAM quota: 1456
[2013-05-20 17:02:39,373] - [xdcrbasetests:910] INFO - Setting XDCR_FAILURE_RESTART_INTERVAL to 1 ..
[2013-05-20 17:02:39,373] - [remote_util:117] INFO - connecting to 10.3.2.144 with username : Administrator password : Membase123 ssh_key: 
[2013-05-20 17:02:44,908] - [remote_util:149] INFO - Connected
[2013-05-20 17:02:45,910] - [remote_util:470] INFO - File /cygdrive/c/tmp/windows_info.txt was found
[2013-05-20 17:02:46,142] - [remote_util:517] INFO - get windows information
[2013-05-20 17:02:46,158] - [remote_util:1264] INFO - running command.raw on 10.3.2.144: hostname
[2013-05-20 17:02:46,560] - [remote_util:1293] INFO - command executed successfully
[2013-05-20 17:03:16,591] - [remote_util:117] INFO - connecting to 10.3.2.244 with username : Administrator password : Membase123 ssh_key: 
[2013-05-20 17:03:22,151] - [remote_util:149] INFO - Connected
[2013-05-20 17:03:23,065] - [remote_util:470] INFO - File /cygdrive/c/tmp/windows_info.txt was found
[2013-05-20 17:03:23,200] - [remote_util:517] INFO - get windows information
[2013-05-20 17:03:23,237] - [remote_util:1264] INFO - running command.raw on 10.3.2.244: hostname
[2013-05-20 17:03:23,574] - [remote_util:1293] INFO - command executed successfully
[2013-05-20 17:03:53,588] - [remote_util:117] INFO - connecting to 10.3.2.250 with username : Administrator password : Membase123 ssh_key: 
[2013-05-20 17:03:59,068] - [remote_util:149] INFO - Connected
[2013-05-20 17:03:59,958] - [remote_util:470] INFO - File /cygdrive/c/tmp/windows_info.txt was found
[2013-05-20 17:04:00,087] - [remote_util:517] INFO - get windows information
[2013-05-20 17:04:00,101] - [remote_util:1264] INFO - running command.raw on 10.3.2.250: hostname
[2013-05-20 17:04:00,401] - [remote_util:1293] INFO - command executed successfully
[2013-05-20 17:04:30,430] - [remote_util:117] INFO - connecting to 10.3.2.125 with username : Administrator password : Membase123 ssh_key: 
[2013-05-20 17:04:35,953] - [remote_util:149] INFO - Connected
[2013-05-20 17:04:36,943] - [remote_util:470] INFO - File /cygdrive/c/tmp/windows_info.txt was found
[2013-05-20 17:04:37,068] - [remote_util:517] INFO - get windows information
[2013-05-20 17:04:37,085] - [remote_util:1264] INFO - running command.raw on 10.3.2.125: hostname
[2013-05-20 17:04:37,420] - [remote_util:1293] INFO - command executed successfully
[2013-05-20 17:05:07,449] - [remote_util:117] INFO - connecting to 10.3.2.123 with username : Administrator password : Membase123 ssh_key: 
[2013-05-20 17:05:12,920] - [remote_util:149] INFO - Connected
[2013-05-20 17:05:14,254] - [remote_util:470] INFO - File /cygdrive/c/tmp/windows_info.txt was found
[2013-05-20 17:05:14,479] - [remote_util:517] INFO - get windows information
[2013-05-20 17:05:14,494] - [remote_util:1264] INFO - running command.raw on 10.3.2.123: hostname
[2013-05-20 17:05:14,922] - [remote_util:1293] INFO - command executed successfully
[2013-05-20 17:05:44,954] - [remote_util:117] INFO - connecting to 10.3.2.124 with username : Administrator password : Membase123 ssh_key: 
[2013-05-20 17:05:50,554] - [remote_util:149] INFO - Connected
[2013-05-20 17:05:51,795] - [remote_util:470] INFO - File /cygdrive/c/tmp/windows_info.txt was found
[2013-05-20 17:05:52,001] - [remote_util:517] INFO - get windows information
[2013-05-20 17:05:52,023] - [remote_util:1264] INFO - running command.raw on 10.3.2.124: hostname
[2013-05-20 17:05:52,483] - [remote_util:1293] INFO - command executed successfully
[2013-05-20 17:06:22,576] - [xdcrbasetests:322] INFO - sleep for 30 secs.  ...
[2013-05-20 17:06:52,635] - [rest_client:731] INFO - adding remote cluster hostname:10.3.2.125:8091 with username:password Administrator:password name:cluster1
[2013-05-20 17:06:53,937] - [rest_client:687] ERROR - http://10.3.2.144:8091/pools/default/remoteClusters error 400 reason: unknown {"_":"Could not connect to \"10.3.2.125\" on port 8091.  This could be due to an incorrect host/port combination or a firewall in place between the servers."}
[2013-05-20 17:06:53,937] - [rest_client:744] ERROR - /remoteCluster failed : status:False,content:{"_":"Could not connect to \"10.3.2.125\" on port 8091.  This could be due to an incorrect host/port combination or a firewall in place between the servers."}
[2013-05-20 17:06:53,938] - [xdcrbasetests:115] ERROR - remoteCluster API 'add cluster' failed
[2013-05-20 17:06:53,938] - [xdcrbasetests:116] ERROR - Error while setting up clusters: (<type 'exceptions.Exception'>, Exception("remoteCluster API 'add cluster' failed",), <traceback object at 0x295b908>)
[2013-05-20 17:06:53,938] - [xdcrbasetests:125] INFO - ==============  XDCRbasetests stats for test #15 load_with_failover ==============
[2013-05-20 17:06:53,939] - [xdcrbasetests:425] INFO - Error while cleaning broken setup.
ERROR