We are supposed to have 14 magic sessions (automatic updates) every hour. It turns out we have exactly on every two hours, always the same: 08:54:04 mohd zahari bin mat amin session (magic) * login: zaharey@gmail.com * userdata file: /home/aras/acis/userdata/z/a/zaharey@gmail.com.xml What is going on? Note that this session never completes. Christian Zimmermann FIGUGEGL! Department of Economics University of Connecticut 341 Mansfield Road, Unit 1063 Storrs, CT 06269-1063 http://ideas.repec.org/zimm/ christian.zimmermann@uconn.edu http://ideas.repec.org/e/pzi1.html
We really have a problem. I logged into this account, but could not do anything in it, not even display its overview. But worse, while there, I was unable to refresh the events log. Somehow, this incapacitated the server, whose load dropped from 3+ to 1. This is all mysterious to me, but clearly needs fixing. On Sat, 14 Jun 2008, Christian Zimmermann wrote:
We are supposed to have 14 magic sessions (automatic updates) every hour. It turns out we have exactly on every two hours, always the same:
08:54:04 mohd zahari bin mat amin session (magic)
* login: zaharey@gmail.com * userdata file: /home/aras/acis/userdata/z/a/zaharey@gmail.com.xml
What is going on? Note that this session never completes.
Christian Zimmermann FIGUGEGL! Department of Economics University of Connecticut 341 Mansfield Road, Unit 1063 Storrs, CT 06269-1063 http://ideas.repec.org/zimm/ christian.zimmermann@uconn.edu http://ideas.repec.org/e/pzi1.html
Christian Zimmermann writes
We really have a problem. I logged into this account, but could not do anything in it, not even display its overview. But worse, while there, I was unable to refresh the events log. Somehow, this incapacitated the server, whose load dropped from 3+ to 1. This is all mysterious to me, but clearly needs fixing.
I will look into this later this evening. One thing that I have changed is data processing. I set the too_old parameter to one year. This means that unchanged files will only be reprocessed after one year, not after one week as it is by default. Changed files will continue to be processed daily. aras@nebka:~$ diff acis/bin/crontab.daily acis/bin/crontab.daily~ 22c22 < /home/aras/acis/bin/updareq RePEc / 31556926 ---
/home/aras/acis/bin/updareq RePEc / 24c24 < /home/aras/acis/bin/updareq ACIS / 31556926
/home/aras/acis/bin/updareq ACIS / 26c26 < /home/aras/acis/bin/updareq citec / 31556926
/home/aras/acis/bin/updareq citec /
Cheers, Thomas Krichel http://openlib.org/home/krichel RePEc:per:1965-06-05:thomas_krichel phone: +7 383 330 6813 skype: thomaskrichel
Good. I notivced in addition that any access to the webserver was incapacitated for about 15 minutes. This happening every two hours is not good. On Sun, 15 Jun 2008, Thomas Krichel wrote:
Christian Zimmermann writes
We really have a problem. I logged into this account, but could not do anything in it, not even display its overview. But worse, while there, I was unable to refresh the events log. Somehow, this incapacitated the server, whose load dropped from 3+ to 1. This is all mysterious to me, but clearly needs fixing.
I will look into this later this evening.
One thing that I have changed is data processing. I set the too_old parameter to one year. This means that unchanged files will only be reprocessed after one year, not after one week as it is by default. Changed files will continue to be processed daily.
aras@nebka:~$ diff acis/bin/crontab.daily acis/bin/crontab.daily~ 22c22 < /home/aras/acis/bin/updareq RePEc / 31556926 ---
/home/aras/acis/bin/updareq RePEc / 24c24 < /home/aras/acis/bin/updareq ACIS / 31556926
/home/aras/acis/bin/updareq ACIS / 26c26 < /home/aras/acis/bin/updareq citec / 31556926
/home/aras/acis/bin/updareq citec /
Cheers,
Thomas Krichel http://openlib.org/home/krichel RePEc:per:1965-06-05:thomas_krichel phone: +7 383 330 6813 skype: thomaskrichel
Christian Zimmermann writes
Good. I notivced in addition that any access to the webserver was incapacitated for about 15 minutes. This happening every two hours is not good.
This has more the whiff of another issue to me. Cheers, Thomas Krichel http://openlib.org/home/krichel RePEc:per:1965-06-05:thomas_krichel phone: +7 383 330 6813 skype: thomaskrichel
Earliest appaerance of the problem, or at least magic sessions for this user: aras@nebka:~/oldlog$ zgrep zaharey back.log.46.gz Tue Apr 29 22:59:28 2008 [24326] for profile pza50 (repec:per:2005-05-09:mr_zaharey_sr_) Tue Apr 29 22:59:28 2008 [24326] exact name: 'mr zaharey sr', found: 0 Tue Apr 29 22:59:28 2008 [24326] exact name: 'zaharey mr sr', found: 0 Tue Apr 29 22:59:28 2008 [24326] exact name: 'm zaharey sr', found: 0 Tue Apr 29 22:59:28 2008 [24326] exact name: 'zaharey sr m', found: 0 Tue Apr 29 22:59:28 2008 [24326] exact name: 'mr zaharey', found: 0 Tue Apr 29 22:59:28 2008 [24326] exact name: 'zaharey mr', found: 0 Tue Apr 29 22:59:28 2008 [24326] exact name: 'm zaharey', found: 0 Tue Apr 29 22:59:28 2008 [24326] exact name: 'zaharey m', found: 0 Note that he has more name variations, so it hangs in the search. He has then magic sessions on a reglar basis ever since. Note that simply removing this account from the magic session schedule may solve the symptom... Christian Zimmermann FIGUGEGL! Department of Economics University of Connecticut 341 Mansfield Road, Unit 1063 Storrs, CT 06269-1063 http://ideas.repec.org/zimm/ christian.zimmermann@uconn.edu http://ideas.repec.org/e/pzi1.html On Sun, 15 Jun 2008, Thomas Krichel wrote:
Christian Zimmermann writes
Good. I notivced in addition that any access to the webserver was incapacitated for about 15 minutes. This happening every two hours is not good.
This has more the whiff of another issue to me.
Cheers,
Thomas Krichel http://openlib.org/home/krichel RePEc:per:1965-06-05:thomas_krichel phone: +7 383 330 6813 skype: thomaskrichel
Christian Zimmermann writes
Note that he has more name variations, so it hangs in the search.
It sholud not, but there may be a timeout, esp if the machine is busy. metadata updates are a killer job.
He has then magic sessions on a reglar basis ever since.
Note that simply removing this account from the magic session schedule may solve the symptom...
Can you do this? I don't know how. I can then try to check him out separately. I am getting very tired. Cheers, Thomas Krichel http://openlib.org/home/krichel RePEc:per:1965-06-05:thomas_krichel phone: +7 383 330 6813 skype: thomaskrichel
I am not sure zaharey is the problem. aras@nebka:apu-job.log suggests Rob Engle is the problem. Cheers, Thomas Krichel http://openlib.org/home/krichel RePEc:per:1965-06-05:thomas_krichel phone: +7 383 330 6813 skype: thomaskrichel
Why him in particular? he appears at the very start of a long log file, and not thereafter. Christian Zimmermann FIGUGEGL! Department of Economics University of Connecticut 341 Mansfield Road, Unit 1063 Storrs, CT 06269-1063 http://ideas.repec.org/zimm/ christian.zimmermann@uconn.edu http://ideas.repec.org/e/pzi1.html On Sun, 15 Jun 2008, Thomas Krichel wrote:
I am not sure zaharey is the problem. aras@nebka:apu-job.log suggests Rob Engle is the problem.
Cheers,
Thomas Krichel http://openlib.org/home/krichel RePEc:per:1965-06-05:thomas_krichel phone: +7 383 330 6813 skype: thomaskrichel
On Sun, 15 Jun 2008, Thomas Krichel wrote:
Christian Zimmermann writes
Note that he has more name variations, so it hangs in the search.
It sholud not, but there may be a timeout, esp if the machine is busy. metadata updates are a killer job.
It is independent of other jobs.
He has then magic sessions on a reglar basis ever since.
Note that simply removing this account from the magic session schedule may solve the symptom...
Can you do this? I don't know how.
Just found this: http://acis.openlib.org/software/doc/apu.html#id2245992 aras@nebka:/var/lib/mysql$ mysql acis -u corner -p Enter password: Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 16028 Server version: 5.0.32-Debian_7etch5-log Debian etch distribution Type 'help;' or '\h' for help. Type '\c' to clear the buffer. mysql> select * from apu_queue where what='pza50'; +-------+------------+---------------------+--------+-------+-------+---------------------+ | what | position | filed | status | class | notes | worked | +-------+------------+---------------------+--------+-------+-------+---------------------+ | pza50 | 1200593767 | 2008-04-22 20:54:05 | | 0 | NULL | 0000-00-00 00:00:00 | +-------+------------+---------------------+--------+-------+-------+---------------------+ 1 row in set (0.23 sec) mysql> select * from apu_queue where what='pzi1'; +------+------------+---------------------+--------+-------+-------+---------------------+ | what | position | filed | status | class | notes | worked | +------+------------+---------------------+--------+-------+-------+---------------------+ | pzi1 | 1206903310 | 2008-04-22 20:54:05 | | 0 | NULL | 0000-00-00 00:00:00 | +------+------------+---------------------+--------+-------+-------+---------------------+ 1 row in set (0.08 sec) mysql> replace into apu_queue (position,what) values('1206903310','pza50'); Query OK, 2 rows affected (0.06 sec) mysql> select * from apu_queue where what='pza50'; +-------+------------+---------------------+--------+-------+-------+---------------------+ | what | position | filed | status | class | notes | worked | +-------+------------+---------------------+--------+-------+-------+---------------------+ | pza50 | 1206903310 | 2008-06-15 11:55:55 | | 0 | NULL | 0000-00-00 00:00:00 | +-------+------------+---------------------+--------+-------+-------+---------------------+ 1 row in set (0.00 sec)
I can then try to check him out separately.
I am getting very tired.
Cheers,
Thomas Krichel http://openlib.org/home/krichel RePEc:per:1965-06-05:thomas_krichel phone: +7 383 330 6813 skype: thomaskrichel
Magic sessions are now working as usual. So it is this particular account that was creating problems: zaharey@gmail.com. Here are his name variations: <names-list> <list-item>mr zaharey sr</list-item> <list-item>zaharey mr sr</list-item> <list-item>m zaharey sr</list-item> <list-item>zaharey sr m</list-item> <list-item>mr zaharey</list-item> <list-item>zaharey mr</list-item> <list-item>m zaharey</list-item> <list-item>zaharey m</list-item> <list-item>economic</list-item> <list-item>finance</list-item> <list-item>journal</list-item> <list-item>myecom my ecommerce com</list-item> <list-item>mohd zahari bin mat amin</list-item> </names-list> It is a rather silly list... But anyway, the script seems to be hanging constant one the "economic" one: Wed Apr 30 19:00:39 2008 [3381] exact name: 'mr zaharey sr', found: 0 Wed Apr 30 19:00:39 2008 [3381] exact name: 'zaharey mr sr', found: 0 Wed Apr 30 19:00:39 2008 [3381] exact name: 'm zaharey sr', found: 0 Wed Apr 30 19:00:39 2008 [3381] exact name: 'zaharey sr m', found: 0 Wed Apr 30 19:00:39 2008 [3381] exact name: 'mr zaharey', found: 0 Wed Apr 30 19:00:39 2008 [3381] exact name: 'zaharey mr', found: 0 Wed Apr 30 19:00:39 2008 [3381] exact name: 'm zaharey', found: 0 Wed Apr 30 19:00:39 2008 [3381] exact name: 'zaharey m', found: 0 Wed Apr 30 20:54:03 2008 [3990] for profile pza50 (repec:per:2005-05-09:mr_zaharey_sr_) Wed Apr 30 20:54:03 2008 [3990] exact name: 'mr zaharey sr', found: 0 Wed Apr 30 20:54:03 2008 [3990] exact name: 'zaharey mr sr', found: 0 Wed Apr 30 20:54:03 2008 [3990] exact name: 'm zaharey sr', found: 0 Wed Apr 30 20:54:03 2008 [3990] exact name: 'zaharey sr m', found: 0 Wed Apr 30 20:54:03 2008 [3990] exact name: 'mr zaharey', found: 0 Wed Apr 30 20:54:03 2008 [3990] exact name: 'zaharey mr', found: 0 Wed Apr 30 20:54:03 2008 [3990] exact name: 'm zaharey', found: 0 Wed Apr 30 20:54:03 2008 [3990] exact name: 'zaharey m', found: 0 Wed Apr 30 21:54:04 2008 [8686] for profile pza50 (repec:per:2005-05-09:mr_zaharey_sr_) Wed Apr 30 21:54:04 2008 [8686] exact name: 'mr zaharey sr', found: 0 Wed Apr 30 21:54:04 2008 [8686] exact name: 'zaharey mr sr', found: 0 Wed Apr 30 21:54:04 2008 [8686] exact name: 'm zaharey sr', found: 0 Wed Apr 30 21:54:04 2008 [8686] exact name: 'zaharey sr m', found: 0 Wed Apr 30 21:54:04 2008 [8686] exact name: 'mr zaharey', found: 0 Wed Apr 30 21:54:04 2008 [8686] exact name: 'zaharey mr', found: 0 Wed Apr 30 21:54:04 2008 [8686] exact name: 'm zaharey', found: 0 Wed Apr 30 21:54:04 2008 [8686] exact name: 'zaharey m', found: 0 Wed Apr 30 22:54:03 2008 [8934] for profile pza50 (repec:per:2005-05-09:mr_zaharey_sr_) Wed Apr 30 22:54:03 2008 [8934] exact name: 'mr zaharey sr', found: 0 Wed Apr 30 22:54:03 2008 [8934] exact name: 'zaharey mr sr', found: 0 Wed Apr 30 22:54:03 2008 [8934] exact name: 'm zaharey sr', found: 0 Wed Apr 30 22:54:03 2008 [8934] exact name: 'zaharey sr m', found: 0 Wed Apr 30 22:54:03 2008 [8934] exact name: 'mr zaharey', found: 0 Wed Apr 30 22:54:03 2008 [8934] exact name: 'zaharey mr', found: 0 Wed Apr 30 22:54:03 2008 [8934] exact name: 'm zaharey', found: 0 Wed Apr 30 22:54:03 2008 [8934] exact name: 'zaharey m', found: 0 etc. Christian Zimmermann FIGUGEGL! Department of Economics University of Connecticut 341 Mansfield Road, Unit 1063 Storrs, CT 06269-1063 http://ideas.repec.org/zimm/ christian.zimmermann@uconn.edu http://ideas.repec.org/e/pzi1.html On Sun, 15 Jun 2008, Christian Zimmermann wrote:
On Sun, 15 Jun 2008, Thomas Krichel wrote:
Christian Zimmermann writes
Note that he has more name variations, so it hangs in the search.
It sholud not, but there may be a timeout, esp if the machine is busy. metadata updates are a killer job.
It is independent of other jobs.
He has then magic sessions on a reglar basis ever since.
Note that simply removing this account from the magic session schedule may solve the symptom...
Can you do this? I don't know how.
Just found this: http://acis.openlib.org/software/doc/apu.html#id2245992
aras@nebka:/var/lib/mysql$ mysql acis -u corner -p Enter password: Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A
Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 16028 Server version: 5.0.32-Debian_7etch5-log Debian etch distribution
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> select * from apu_queue where what='pza50'; +-------+------------+---------------------+--------+-------+-------+---------------------+ | what | position | filed | status | class | notes | worked | +-------+------------+---------------------+--------+-------+-------+---------------------+ | pza50 | 1200593767 | 2008-04-22 20:54:05 | | 0 | NULL | 0000-00-00 00:00:00 | +-------+------------+---------------------+--------+-------+-------+---------------------+ 1 row in set (0.23 sec)
mysql> select * from apu_queue where what='pzi1'; +------+------------+---------------------+--------+-------+-------+---------------------+ | what | position | filed | status | class | notes | worked | +------+------------+---------------------+--------+-------+-------+---------------------+ | pzi1 | 1206903310 | 2008-04-22 20:54:05 | | 0 | NULL | 0000-00-00 00:00:00 | +------+------------+---------------------+--------+-------+-------+---------------------+ 1 row in set (0.08 sec)
mysql> replace into apu_queue (position,what) values('1206903310','pza50'); Query OK, 2 rows affected (0.06 sec)
mysql> select * from apu_queue where what='pza50'; +-------+------------+---------------------+--------+-------+-------+---------------------+ | what | position | filed | status | class | notes | worked | +-------+------------+---------------------+--------+-------+-------+---------------------+ | pza50 | 1206903310 | 2008-06-15 11:55:55 | | 0 | NULL | 0000-00-00 00:00:00 | +-------+------------+---------------------+--------+-------+-------+---------------------+ 1 row in set (0.00 sec)
I can then try to check him out separately.
I am getting very tired.
Cheers,
Thomas Krichel http://openlib.org/home/krichel RePEc:per:1965-06-05:thomas_krichel phone: +7 383 330 6813 skype: thomaskrichel
_______________________________________________ RAS-run mailing list RAS-run@lists.openlib.org http://lists.openlib.org/cgi-bin/mailman/listinfo/ras-run
Christian Zimmermann writes
It is a rather silly list... But anyway, the script seems to be hanging constant one the "economic" one:
And the apu-log shows "out of memory". It is the memory leak in the sql_helper.pm. Cheers, Thomas Krichel http://openlib.org/home/krichel RePEc:per:1965-06-05:thomas_krichel phone: +7 383 330 6813 skype: thomaskrichel
The problem reappeared with the supposed memory leak. Same account. I pushed its refresh date further in the future. Christian Zimmermann FIGUGEGL! Department of Economics University of Connecticut 341 Mansfield Road, Unit 1063 Storrs, CT 06269-1063 http://ideas.repec.org/zimm/ christian.zimmermann@uconn.edu http://ideas.repec.org/e/pzi1.html On Sun, 15 Jun 2008, Christian Zimmermann wrote:
On Sun, 15 Jun 2008, Thomas Krichel wrote:
Christian Zimmermann writes
Note that he has more name variations, so it hangs in the search.
It sholud not, but there may be a timeout, esp if the machine is busy. metadata updates are a killer job.
It is independent of other jobs.
He has then magic sessions on a reglar basis ever since.
Note that simply removing this account from the magic session schedule may solve the symptom...
Can you do this? I don't know how.
Just found this: http://acis.openlib.org/software/doc/apu.html#id2245992
aras@nebka:/var/lib/mysql$ mysql acis -u corner -p Enter password: Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A
Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 16028 Server version: 5.0.32-Debian_7etch5-log Debian etch distribution
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql> select * from apu_queue where what='pza50'; +-------+------------+---------------------+--------+-------+-------+---------------------+ | what | position | filed | status | class | notes | worked | +-------+------------+---------------------+--------+-------+-------+---------------------+ | pza50 | 1200593767 | 2008-04-22 20:54:05 | | 0 | NULL | 0000-00-00 00:00:00 | +-------+------------+---------------------+--------+-------+-------+---------------------+ 1 row in set (0.23 sec)
mysql> select * from apu_queue where what='pzi1'; +------+------------+---------------------+--------+-------+-------+---------------------+ | what | position | filed | status | class | notes | worked | +------+------------+---------------------+--------+-------+-------+---------------------+ | pzi1 | 1206903310 | 2008-04-22 20:54:05 | | 0 | NULL | 0000-00-00 00:00:00 | +------+------------+---------------------+--------+-------+-------+---------------------+ 1 row in set (0.08 sec)
mysql> replace into apu_queue (position,what) values('1206903310','pza50'); Query OK, 2 rows affected (0.06 sec)
mysql> select * from apu_queue where what='pza50'; +-------+------------+---------------------+--------+-------+-------+---------------------+ | what | position | filed | status | class | notes | worked | +-------+------------+---------------------+--------+-------+-------+---------------------+ | pza50 | 1206903310 | 2008-06-15 11:55:55 | | 0 | NULL | 0000-00-00 00:00:00 | +-------+------------+---------------------+--------+-------+-------+---------------------+ 1 row in set (0.00 sec)
I can then try to check him out separately.
I am getting very tired.
Cheers,
Thomas Krichel http://openlib.org/home/krichel RePEc:per:1965-06-05:thomas_krichel phone: +7 383 330 6813 skype: thomaskrichel
Christian Zimmermann writes
The problem reappeared with the supposed memory leak. Same account. I pushed its refresh date further in the future.
I have spent some time working on a version of the library that does not have the memory hug. But I went back to the original version with the last bug on the background mysql queries. In any case, introducing a fix is real risky because the module is used all over the place in ACIS. Cheers, Thomas Krichel http://openlib.org/home/krichel RePEc:per:1965-06-05:thomas_krichel skype: thomaskrichel
participants (2)
-
Christian Zimmermann -
Thomas Krichel