About initLimit and syncLimit

Discussion:

charles charles

2012-09-24 13:25:53 UTC

initLimitAmount of time, in ticks (see tickTime), to allow followers to connect and sync to a leader. Increased this value as needed, if the amount of data managed by ZooKeeper is large.syncLimitAmount of time, in ticks (see tickTime), to allow followers to sync with ZooKeeper. If followers fall too far behind a leader, they will be dropped. In zookeeper config file zoo.cfg, we can config above two parameters, but just from above description, I can't know the difference between initLimit and syncLimit. Anyone can help me, tell me more about them, since my server can't come up, maybe, reson in in these two parameters. Thx! Charles

charles charles

2012-09-24 13:30:48 UTC

Permalink

initLimit
Amount of time, in ticks (see tickTime), to allow followers to connect and sync to a leader. Increased this value as needed, if the amount of data managed by ZooKeeper is large.

syncLimit
Amount of time, in ticks (see tickTime), to allow followers to sync with ZooKeeper. If followers fall too far behind a leader, they will be dropped. In zookeeper config file zoo.cfg, we can config above two parameters, but just from above description, I can't know the difference between initLimit and syncLimit. Anyone can help me, tell me more about them, since my server can't come up, maybe, reson in in these two parameters.

Thx! Charles

Subject: About initLimit and syncLimit
Date: Mon, 24 Sep 2012 06:25:53 -0700
initLimitAmount of time, in ticks (see tickTime), to allow followers to connect and sync to a leader. Increased this value as needed, if the amount of data managed by ZooKeeper is large.syncLimitAmount of time, in ticks (see tickTime), to allow followers to sync with ZooKeeper. If followers fall too far behind a leader, they will be dropped. In zookeeper config file zoo.cfg, we can config above two parameters, but just from above description, I can't know the difference between initLimit and syncLimit. Anyone can help me, tell me more about them, since my server can't come up, maybe, reson in in these two parameters. Thx! Charles

Kathleen Ting

2012-09-24 15:19:00 UTC

Permalink

Charles,

initLimit measures how much time we anticipate the bootstrapping to
need whereas syncLimit controls how far behind the Followers can fall
behind the Leader. Sounds like you might be hitting ZOOKEEPER-1521. As
a result, can you try increasing both initLimit and syncLimit on all
ZK servers and then restarting?

Regards, Kathleen

On Mon, Sep 24, 2012 at 8:30 AM, charles charles

Post by charles charles
initLimit
Amount of time, in ticks (see tickTime), to allow followers to connect and sync to a leader. Increased this value as needed, if the amount of data managed by ZooKeeper is large.
syncLimit
Amount of time, in ticks (see tickTime), to allow followers to sync with ZooKeeper. If followers fall too far behind a leader, they will be dropped. In zookeeper config file zoo.cfg, we can config above two parameters, but just from above description, I can't know the difference between initLimit and syncLimit. Anyone can help me, tell me more about them, since my server can't come up, maybe, reson in in these two parameters.
Thx! Charles

charles charles

2012-09-24 23:52:14 UTC

Permalink

Thx, kathleen. This is my understanding. If I config initLimit=10 and syncLimit=3, so as follow: 1. Leader will wait for follow 10*tickTime to do such things: connect to leader, and sync the lasted data from leader. But, if the time follow load snap data from local disk is included in 10*tickTime?

Load snap data from local disk

Conn to leader

Sync lasted data from leader

A: load snap data from local disk,B: conn to leaderC: sync lasted data from leader. Maybe, you can tell me, 10*tickTime includes whick part? A or B+C or A+B+C 2. While in runtime, if follow's state too far behind a leadr, even to 3*tickTime, the follow will be dropped. Others, please check my understanding, thx! Charles

Date: Mon, 24 Sep 2012 10:19:00 -0500
Subject: Re: About initLimit and syncLimit
Charles,
initLimit measures how much time we anticipate the bootstrapping to
need whereas syncLimit controls how far behind the Followers can fall
behind the Leader. Sounds like you might be hitting ZOOKEEPER-1521. As
a result, can you try increasing both initLimit and syncLimit on all
ZK servers and then restarting?
Regards, Kathleen
On Mon, Sep 24, 2012 at 8:30 AM, charles charles

charles charles

2012-09-24 23:59:22 UTC

Permalink

Oh$B!$(BI'm sorry for my dirty format! Thx, kathleen. This is my understanding. If I config initLimit=10 and syncLimit=3, so as follow 2 point: 1. Leader will wait for follow 10*tickTime to do such things: connect to leader, and sync the lasted data from leader. But, if the time follow load snap data from local disk is included in 10*tickTime? A. Load snap data from local disk-->B.Conn to leader-->C. Sync lasted data from leader Maybe, you can tell me, 10*tickTime includes whick part? A or B+C or A+B+C 2. While in runtime, if follow's state too far behind a leadr, even to 3*tickTime, the follow will be dropped. Others, please check my understanding, thx! Charles

Subject: RE: About initLimit and syncLimit
Date: Mon, 24 Sep 2012 16:52:14 -0700
Thx, kathleen. This is my understanding. If I config initLimit=10 and syncLimit=3, so as follow: 1. Leader will wait for follow 10*tickTime to do such things: connect to leader, and sync the lasted data from leader. But, if the time follow load snap data from local disk is included in 10*tickTime?
Load snap data from local disk
Conn to leader
Sync lasted data from leader
A: load snap data from local disk,B: conn to leaderC: sync lasted data from leader. Maybe, you can tell me, 10*tickTime includes whick part? A or B+C or A+B+C 2. While in runtime, if follow's state too far behind a leadr, even to 3*tickTime, the follow will be dropped. Others, please check my understanding, thx! Charles

ramachandra

2012-09-25 10:48:47 UTC

Permalink

Hi,

Even I am facing this issue. But I dont want to clean up the data
directory, Is there any other way of solving this issue?? Please let me know
as soon as possible.

Regards,
Ramachandra

--
View this message in context: http://zookeeper-user.578899.n2.nabble.com/Re-Zookeeper-cluster-failing-tp7577972p7577980.html
Sent from the zookeeper-user mailing list archive at Nabble.com.

charles charles

2012-10-04 09:17:45 UTC

Permalink

Yeah, I maybe the same problem! Maybe we misunderstand the initLimit and syncLimit.Please check for me: 1. Leader will wait for follow 10*tickTime to do such things: connect to leader, and sync the lasted data from leader. But, if the time follow load snap data from local disk is included in 10*tickTime?
A: load snap data from local disk,
B: conn to leader
C: sync lasted data from leader.

Maybe, you can tell me, 10*tickTime includes which part?
A or B+C or A+B+C

2. While in runtime, if follow's state too far behind a leadr, even to 3*tickTime, the follow will be dropped.

Others, please check my understanding

Date: Tue, 25 Sep 2012 03:48:47 -0700
Subject: RE: About initLimit and syncLimit
Hi,
Even I am facing this issue. But I dont want to clean up the data
directory, Is there any other way of solving this issue?? Please let me know
as soon as possible.
Regards,
Ramachandra
--
View this message in context: http://zookeeper-user.578899.n2.nabble.com/Re-Zookeeper-cluster-failing-tp7577972p7577980.html
Sent from the zookeeper-user mailing list archive at Nabble.com.

Patrick Hunt

2012-10-06 14:41:16 UTC

Permalink

Say you have a very large datastore (the snapshot file is very large).
It can take considerable time for the leader to communicate the znode
data to the followers during startup. initLimit is the amount of time
the leader will allow the followers to d/l the data from the leader
and get to a point where they are ready to server requests to clients.
If this time is exceeded the leader will close the connection to the
follower and the quorum process will restart. 20 sec is typically more
than enough time, however if you have a slow connection or the data is
very large it might be necessary to increase this. For example I've
seen some cases of users storing 100's of gb in their datastore, in
which case the initLimit needed to be increased. However in most cases
this is not necessary.

The syncLimit controls the amount of time the quorum members will
allow each other to communicate. If the leader doesn't hear from a
follower in the syncLimit time it will drop the follower from the
quorum, and vice versa. It's important not to set this parameter too
high as it's one of the ways a server detects networking issues and
causes recovery to take place (ie the follower will drop out of the
quorum and try to reconnect, all clients on that follower will
disconnect and reconnect to another server, etc...)

Patrick

Post by charles charles
Yeah, I maybe the same problem! Maybe we misunderstand the initLimit and syncLimit.Please check for me: 1. Leader will wait for follow 10*tickTime to do such things: connect to leader, and sync the lasted data from leader. But, if the time follow load snap data from local disk is included in 10*tickTime?
A: load snap data from local disk,
B: conn to leader
C: sync lasted data from leader.
Maybe, you can tell me, 10*tickTime includes which part?
A or B+C or A+B+C
2. While in runtime, if follow's state too far behind a leadr, even to 3*tickTime, the follow will be dropped.
Others, please check my understanding

Patrick Hunt

2012-10-06 14:41:16 UTC

Permalink

charles charles

2012-10-09 11:49:12 UTC

Permalink

Ok,thank to phunt!

Date: Sat, 6 Oct 2012 07:41:16 -0700
Subject: Re: About initLimit and syncLimit
Say you have a very large datastore (the snapshot file is very large).
It can take considerable time for the leader to communicate the znode
data to the followers during startup. initLimit is the amount of time
the leader will allow the followers to d/l the data from the leader
and get to a point where they are ready to server requests to clients.
If this time is exceeded the leader will close the connection to the
follower and the quorum process will restart. 20 sec is typically more
than enough time, however if you have a slow connection or the data is
very large it might be necessary to increase this. For example I've
seen some cases of users storing 100's of gb in their datastore, in
which case the initLimit needed to be increased. However in most cases
this is not necessary.
The syncLimit controls the amount of time the quorum members will
allow each other to communicate. If the leader doesn't hear from a
follower in the syncLimit time it will drop the follower from the
quorum, and vice versa. It's important not to set this parameter too
high as it's one of the ways a server detects networking issues and
causes recovery to take place (ie the follower will drop out of the
quorum and try to reconnect, all clients on that follower will
disconnect and reconnect to another server, etc...)
Patrick