./testrunner -i /tmp/6-win-xdcr.ini get-cbcollect-info=True -t xdcr.biXDCR.bidirectional.load_with_failover,replicas=1,items=10000,ctopology=chain,rdirection=bidirection,doc-ops=create-update-delete,doc-ops-dest=create,update,failover=source Test Input params: {'case_number': 15, 'cluster_name': '6-win-xdcr', 'conf_file': '/tmp/six-node-xdcr.conf', 'ctopology': 'chain', 'doc-ops': 'create-update-delete', 'doc-ops-dest': 'create,update', 'failover': 'source', 'get-cbcollect-info': 'True', 'ini': '/tmp/6-win-xdcr.ini', 'items': '10000', 'num_nodes': 6, 'rdirection': 'bidirection', 'replicas': '1', 'spec': 'six-node-xdcr'} [2013-05-20 17:01:49,465] - [xdcrbasetests:204] INFO - Initializing input parameters started... [2013-05-20 17:01:49,466] - [xdcrbasetests:301] INFO - Initializing input parameters completed. [2013-05-20 17:01:49,468] - [xdcrbasetests:84] INFO - ============== XDCRbasetests setup was started for test #15 load_with_failover============== [2013-05-20 17:01:49,527] - [xdcrbasetests:408] INFO - cleanup cluster1: [ip:10.3.2.144 port:8091 ssh_username:Administrator, ip:10.3.2.244 port:8091 ssh_username:Administrator, ip:10.3.2.250 port:8091 ssh_username:Administrator] [2013-05-20 17:01:49,544] - [bucket_helper:136] INFO - deleting existing buckets [] on 10.3.2.144 [2013-05-20 17:01:49,599] - [cluster_helper:84] INFO - waiting for ns_server @ 10.3.2.144:8091 [2013-05-20 17:01:49,611] - [cluster_helper:86] INFO - ns_server @ 10.3.2.144:8091 is running [2013-05-20 17:01:49,626] - [bucket_helper:136] INFO - deleting existing buckets [] on 10.3.2.244 [2013-05-20 17:01:49,670] - [cluster_helper:84] INFO - waiting for ns_server @ 10.3.2.244:8091 [2013-05-20 17:01:49,679] - [cluster_helper:86] INFO - ns_server @ 10.3.2.244:8091 is running [2013-05-20 17:01:49,692] - [bucket_helper:136] INFO - deleting existing buckets [] on 10.3.2.250 [2013-05-20 17:01:49,733] - [cluster_helper:84] INFO - waiting for ns_server @ 10.3.2.250:8091 [2013-05-20 17:01:49,741] - [cluster_helper:86] INFO - ns_server @ 10.3.2.250:8091 is running [2013-05-20 17:01:49,742] - [xdcrbasetests:408] INFO - cleanup cluster2: [ip:10.3.2.125 port:8091 ssh_username:Administrator, ip:10.3.2.123 port:8091 ssh_username:Administrator, ip:10.3.2.124 port:8091 ssh_username:Administrator] [2013-05-20 17:01:49,758] - [bucket_helper:136] INFO - deleting existing buckets [] on 10.3.2.125 [2013-05-20 17:01:49,804] - [cluster_helper:84] INFO - waiting for ns_server @ 10.3.2.125:8091 [2013-05-20 17:01:49,814] - [cluster_helper:86] INFO - ns_server @ 10.3.2.125:8091 is running [2013-05-20 17:01:49,831] - [bucket_helper:136] INFO - deleting existing buckets [] on 10.3.2.123 [2013-05-20 17:01:49,867] - [cluster_helper:84] INFO - waiting for ns_server @ 10.3.2.123:8091 [2013-05-20 17:01:49,876] - [cluster_helper:86] INFO - ns_server @ 10.3.2.123:8091 is running [2013-05-20 17:01:49,890] - [bucket_helper:136] INFO - deleting existing buckets [] on 10.3.2.124 [2013-05-20 17:01:49,928] - [cluster_helper:84] INFO - waiting for ns_server @ 10.3.2.124:8091 [2013-05-20 17:01:49,937] - [cluster_helper:86] INFO - ns_server @ 10.3.2.124:8091 is running [2013-05-20 17:01:50,682] - [rest_client:1488] INFO - Consistent-views during rebalance was set as indexAwareRebalanceDisabled=true [2013-05-20 17:01:50,683] - [rest_client:704] INFO - settings/web params on 10.3.2.144:8091 :username=Administrator&password=password&port=8091 [2013-05-20 17:01:50,910] - [rest_client:724] INFO - pools/default params : memoryQuota=2184&username=Administrator&password=password [2013-05-20 17:01:51,330] - [rest_client:1488] INFO - Consistent-views during rebalance was set as indexAwareRebalanceDisabled=true [2013-05-20 17:01:51,331] - [rest_client:704] INFO - settings/web params on 10.3.2.244:8091 :username=Administrator&password=password&port=8091 [2013-05-20 17:01:51,565] - [rest_client:724] INFO - pools/default params : memoryQuota=2184&username=Administrator&password=password [2013-05-20 17:01:51,984] - [rest_client:1488] INFO - Consistent-views during rebalance was set as indexAwareRebalanceDisabled=true [2013-05-20 17:01:51,984] - [rest_client:704] INFO - settings/web params on 10.3.2.250:8091 :username=Administrator&password=password&port=8091 [2013-05-20 17:01:52,214] - [rest_client:724] INFO - pools/default params : memoryQuota=2184&username=Administrator&password=password [2013-05-20 17:01:53,422] - [task:260] INFO - adding node 10.3.2.244:8091 to cluster [2013-05-20 17:01:53,422] - [rest_client:839] INFO - adding remote node @10.3.2.244:8091 to this cluster @10.3.2.144:8091 [2013-05-20 17:01:54,256] - [task:260] INFO - adding node 10.3.2.250:8091 to cluster [2013-05-20 17:01:54,257] - [rest_client:839] INFO - adding remote node @10.3.2.250:8091 to this cluster @10.3.2.144:8091 [2013-05-20 17:01:55,254] - [rest_client:925] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.3.2.250%2Cns_1%4010.3.2.144%2Cns_1%4010.3.2.244 [2013-05-20 17:01:55,463] - [rest_client:929] INFO - rebalance operation started [2013-05-20 17:01:55,486] - [rest_client:1031] INFO - rebalance percentage : 0 % [2013-05-20 17:02:05,508] - [task:340] INFO - rebalancing was completed with progress: 100% in 10.0440838337 sec [2013-05-20 17:02:06,521] - [rest_client:1294] INFO - http://10.3.2.144:8091/pools/default/buckets with param: proxyPort=11211&bucketType=membase&authType=sasl&replicaIndex=1&name=default&saslPassword=None&flushEnabled=1&replicaNumber=1&ramQuotaMB=1456 [2013-05-20 17:02:06,753] - [rest_client:1300] INFO - 0.231631994247 seconds to create bucket default [2013-05-20 17:02:06,754] - [bucket_helper:292] INFO - waiting for memcached bucket : default in 10.3.2.144 to accept set ops [2013-05-20 17:02:08,567] - [data_helper:289] INFO - creating direct client 10.3.2.144:11210 default [2013-05-20 17:02:08,980] - [data_helper:289] INFO - creating direct client 10.3.2.244:11210 default [2013-05-20 17:02:09,601] - [data_helper:289] INFO - creating direct client 10.3.2.250:11210 default [2013-05-20 17:02:10,494] - [data_helper:289] INFO - creating direct client 10.3.2.144:11210 default [2013-05-20 17:02:11,023] - [data_helper:289] INFO - creating direct client 10.3.2.244:11210 default [2013-05-20 17:02:11,879] - [data_helper:289] INFO - creating direct client 10.3.2.250:11210 default [2013-05-20 17:02:13,858] - [task:182] INFO - bucket 'default' was created with per node RAM quota: 1456 [2013-05-20 17:02:15,075] - [rest_client:1488] INFO - Consistent-views during rebalance was set as indexAwareRebalanceDisabled=true [2013-05-20 17:02:15,075] - [rest_client:704] INFO - settings/web params on 10.3.2.125:8091 :username=Administrator&password=password&port=8091 [2013-05-20 17:02:15,298] - [rest_client:724] INFO - pools/default params : memoryQuota=2184&username=Administrator&password=password [2013-05-20 17:02:15,720] - [rest_client:1488] INFO - Consistent-views during rebalance was set as indexAwareRebalanceDisabled=true [2013-05-20 17:02:15,721] - [rest_client:704] INFO - settings/web params on 10.3.2.123:8091 :username=Administrator&password=password&port=8091 [2013-05-20 17:02:15,948] - [rest_client:724] INFO - pools/default params : memoryQuota=2184&username=Administrator&password=password [2013-05-20 17:02:16,366] - [rest_client:1488] INFO - Consistent-views during rebalance was set as indexAwareRebalanceDisabled=true [2013-05-20 17:02:16,366] - [rest_client:704] INFO - settings/web params on 10.3.2.124:8091 :username=Administrator&password=password&port=8091 [2013-05-20 17:02:16,585] - [rest_client:724] INFO - pools/default params : memoryQuota=2184&username=Administrator&password=password [2013-05-20 17:02:17,790] - [task:260] INFO - adding node 10.3.2.123:8091 to cluster [2013-05-20 17:02:17,790] - [rest_client:839] INFO - adding remote node @10.3.2.123:8091 to this cluster @10.3.2.125:8091 [2013-05-20 17:02:18,563] - [task:260] INFO - adding node 10.3.2.124:8091 to cluster [2013-05-20 17:02:18,563] - [rest_client:839] INFO - adding remote node @10.3.2.124:8091 to this cluster @10.3.2.125:8091 [2013-05-20 17:02:19,848] - [rest_client:925] INFO - rebalance params : password=password&ejectedNodes=&user=Administrator&knownNodes=ns_1%4010.3.2.125%2Cns_1%4010.3.2.124%2Cns_1%4010.3.2.123 [2013-05-20 17:02:20,054] - [rest_client:929] INFO - rebalance operation started [2013-05-20 17:02:20,063] - [rest_client:1031] INFO - rebalance percentage : 0 % [2013-05-20 17:02:30,087] - [task:340] INFO - rebalancing was completed with progress: 100% in 10.0319011211 sec [2013-05-20 17:02:31,099] - [rest_client:1294] INFO - http://10.3.2.125:8091/pools/default/buckets with param: proxyPort=11211&bucketType=membase&authType=sasl&replicaIndex=1&name=default&saslPassword=None&flushEnabled=1&replicaNumber=1&ramQuotaMB=1456 [2013-05-20 17:02:31,319] - [rest_client:1300] INFO - 0.219950914383 seconds to create bucket default [2013-05-20 17:02:31,320] - [bucket_helper:292] INFO - waiting for memcached bucket : default in 10.3.2.125 to accept set ops [2013-05-20 17:02:33,457] - [data_helper:289] INFO - creating direct client 10.3.2.123:11210 default [2013-05-20 17:02:33,884] - [data_helper:289] INFO - creating direct client 10.3.2.124:11210 default [2013-05-20 17:02:34,549] - [data_helper:289] INFO - creating direct client 10.3.2.125:11210 default [2013-05-20 17:02:35,235] - [data_helper:289] INFO - creating direct client 10.3.2.123:11210 default [2013-05-20 17:02:36,030] - [data_helper:289] INFO - creating direct client 10.3.2.124:11210 default [2013-05-20 17:02:36,709] - [data_helper:289] INFO - creating direct client 10.3.2.125:11210 default [2013-05-20 17:02:39,370] - [task:182] INFO - bucket 'default' was created with per node RAM quota: 1456 [2013-05-20 17:02:39,373] - [xdcrbasetests:910] INFO - Setting XDCR_FAILURE_RESTART_INTERVAL to 1 .. [2013-05-20 17:02:39,373] - [remote_util:117] INFO - connecting to 10.3.2.144 with username : Administrator password : Membase123 ssh_key: [2013-05-20 17:02:44,908] - [remote_util:149] INFO - Connected [2013-05-20 17:02:45,910] - [remote_util:470] INFO - File /cygdrive/c/tmp/windows_info.txt was found [2013-05-20 17:02:46,142] - [remote_util:517] INFO - get windows information [2013-05-20 17:02:46,158] - [remote_util:1264] INFO - running command.raw on 10.3.2.144: hostname [2013-05-20 17:02:46,560] - [remote_util:1293] INFO - command executed successfully [2013-05-20 17:03:16,591] - [remote_util:117] INFO - connecting to 10.3.2.244 with username : Administrator password : Membase123 ssh_key: [2013-05-20 17:03:22,151] - [remote_util:149] INFO - Connected [2013-05-20 17:03:23,065] - [remote_util:470] INFO - File /cygdrive/c/tmp/windows_info.txt was found [2013-05-20 17:03:23,200] - [remote_util:517] INFO - get windows information [2013-05-20 17:03:23,237] - [remote_util:1264] INFO - running command.raw on 10.3.2.244: hostname [2013-05-20 17:03:23,574] - [remote_util:1293] INFO - command executed successfully [2013-05-20 17:03:53,588] - [remote_util:117] INFO - connecting to 10.3.2.250 with username : Administrator password : Membase123 ssh_key: [2013-05-20 17:03:59,068] - [remote_util:149] INFO - Connected [2013-05-20 17:03:59,958] - [remote_util:470] INFO - File /cygdrive/c/tmp/windows_info.txt was found [2013-05-20 17:04:00,087] - [remote_util:517] INFO - get windows information [2013-05-20 17:04:00,101] - [remote_util:1264] INFO - running command.raw on 10.3.2.250: hostname [2013-05-20 17:04:00,401] - [remote_util:1293] INFO - command executed successfully [2013-05-20 17:04:30,430] - [remote_util:117] INFO - connecting to 10.3.2.125 with username : Administrator password : Membase123 ssh_key: [2013-05-20 17:04:35,953] - [remote_util:149] INFO - Connected [2013-05-20 17:04:36,943] - [remote_util:470] INFO - File /cygdrive/c/tmp/windows_info.txt was found [2013-05-20 17:04:37,068] - [remote_util:517] INFO - get windows information [2013-05-20 17:04:37,085] - [remote_util:1264] INFO - running command.raw on 10.3.2.125: hostname [2013-05-20 17:04:37,420] - [remote_util:1293] INFO - command executed successfully [2013-05-20 17:05:07,449] - [remote_util:117] INFO - connecting to 10.3.2.123 with username : Administrator password : Membase123 ssh_key: [2013-05-20 17:05:12,920] - [remote_util:149] INFO - Connected [2013-05-20 17:05:14,254] - [remote_util:470] INFO - File /cygdrive/c/tmp/windows_info.txt was found [2013-05-20 17:05:14,479] - [remote_util:517] INFO - get windows information [2013-05-20 17:05:14,494] - [remote_util:1264] INFO - running command.raw on 10.3.2.123: hostname [2013-05-20 17:05:14,922] - [remote_util:1293] INFO - command executed successfully [2013-05-20 17:05:44,954] - [remote_util:117] INFO - connecting to 10.3.2.124 with username : Administrator password : Membase123 ssh_key: [2013-05-20 17:05:50,554] - [remote_util:149] INFO - Connected [2013-05-20 17:05:51,795] - [remote_util:470] INFO - File /cygdrive/c/tmp/windows_info.txt was found [2013-05-20 17:05:52,001] - [remote_util:517] INFO - get windows information [2013-05-20 17:05:52,023] - [remote_util:1264] INFO - running command.raw on 10.3.2.124: hostname [2013-05-20 17:05:52,483] - [remote_util:1293] INFO - command executed successfully [2013-05-20 17:06:22,576] - [xdcrbasetests:322] INFO - sleep for 30 secs. ... [2013-05-20 17:06:52,635] - [rest_client:731] INFO - adding remote cluster hostname:10.3.2.125:8091 with username:password Administrator:password name:cluster1 [2013-05-20 17:06:53,937] - [rest_client:687] ERROR - http://10.3.2.144:8091/pools/default/remoteClusters error 400 reason: unknown {"_":"Could not connect to \"10.3.2.125\" on port 8091. This could be due to an incorrect host/port combination or a firewall in place between the servers."} [2013-05-20 17:06:53,937] - [rest_client:744] ERROR - /remoteCluster failed : status:False,content:{"_":"Could not connect to \"10.3.2.125\" on port 8091. This could be due to an incorrect host/port combination or a firewall in place between the servers."} [2013-05-20 17:06:53,938] - [xdcrbasetests:115] ERROR - remoteCluster API 'add cluster' failed [2013-05-20 17:06:53,938] - [xdcrbasetests:116] ERROR - Error while setting up clusters: (, Exception("remoteCluster API 'add cluster' failed",), ) [2013-05-20 17:06:53,938] - [xdcrbasetests:125] INFO - ============== XDCRbasetests stats for test #15 load_with_failover ============== [2013-05-20 17:06:53,939] - [xdcrbasetests:425] INFO - Error while cleaning broken setup. ERROR