March 2008 - RAS-run - RePEc Lists

Cron <aras@nebka> /home/aras/acis/bin/crontab.hourly (fwd)
by Christian Zimmermann 19 Mar '08

19 Mar '08

What's this server gone away business? ---------- Forwarded message ---------- Date: Wed, 19 Mar 2008 09:58:34 -0400 From: Cron Daemon <root(a)nebka.openlib.org> To: aras(a)nebka.openlib.org Subject: Cron <aras@nebka> /home/aras/acis/bin/crontab.hourly [28765] Forking a request: ACIS: k/a/katranid(a)uom.gr.xml [28765] Forking a request: ACIS: h/b/hbachev(a)yahoo.com.xml [28765] Forking a request: ACIS: w/i/witzanyj(a)vse.cz.xml [28765] Forking a request: ACIS: j/a/james.davidson(a)exeter.ac.uk.xml [28765] Forking a request: ACIS: m/i/mito(a)decon.edu.uy.xml [28765] Forking a request: ACIS: d/a/danziger(a)bgu.ac.il.xml [28765] Forking a request: ACIS: d/a/danutza(a)utgjiu.ro.xml [28765] Forking a request: ACIS: j/e/jean-luc.prigent(a)u-cergy.fr.xml [28765] Forking a request: ACIS: k/a/karagian(a)uom.gr.xml DBD::mysql::st execute failed: MySQL server has gone away at /home/aras/acis/lib/sql_helper.pm line 411. ...propagated at /home/aras/acis/lib/sql_helper.pm line 289.

2 1

RAS problem
by Christian Zimmermann 18 Mar '08

18 Mar '08

authors.repec.org is generating internal server errors, but I have no way to figure out what is wrong! Christian Zimmermann FIGUGEGL! Department of Economics University of Connecticut 341 Mansfield Road, Unit 1063 Storrs, CT 06269-1063 http://ideas.repec.org/zimm/ christian.zimmermann(a)uconn.edu http://ideas.repec.org/e/pzi1.html

2 4

nebka crontab
by Christian Zimmermann 18 Mar '08

18 Mar '08

I have further worked on the regular jobs for the aras account on nebka. I have put all hourly jobs in one script, to prevent them from colliding. In particular, I found a job that was running every 9 minutes that was touching 42000 files. That is insane. It is now hourly. Hourly jobs run every hours at the 54, except 23:54 (to prevent collision with daily jobs) and 7:54 (to prevent collision with anacron). Christian Zimmermann FIGUGEGL! Department of Economics University of Connecticut 341 Mansfield Road, Unit 1063 Storrs, CT 06269-1063 http://ideas.repec.org/zimm/ christian.zimmermann(a)uconn.edu http://ideas.repec.org/e/pzi1.html

2 2

(no subject)
by Christian Zimmermann 16 Mar '08

16 Mar '08

I am on campus. On my office machine, where I always keep a window on nebka, it says: Message from syslogd@nebka at Sat Mar 15 07:37:41 2008 ... nebka kernel: journal commit I/O error Read from remote host nebka.uconn.edu: Connection timed out Connection to nebka.uconn.edu closed. On nebka itself, I have error messages I have not seen before. I am not putting everything, as I have to copy by hand while switching screens: EIP is at io_callback+0x1c.0xde [aacraid] ... Process swapper (pid:0 ti:c0314000 task:c02c76a0 taks.ti:c0314000) ... Call trace aac_response_normal [aacraid] aac_rx_intr [aacraid] handle_IRQ_event __do_IRQ do_IRQ common_interrupt default_idle default_idle cpu_idle start_kernel EIP: [<f88ad16.] io_callback+0x1d/0xde [aacraid] SS:ESP 0068:c0315f1c <0> kernel panic - not syncing: Fatal exception in interrupt. According to syslog, the last instruction before panic was make-repec-per.sh. At the same time, the daily jobs from anacron were running (last one looks like sysstat/sa1), possibly also awstats.pl (why???) Note that according to crontab for aras, every hour at 37, there is also events_archiving running. I commented it out, as this seems to coincide with the panic. All in all: a lot of things running at the same time, all with lots of I/O. If on top of that there in a rsync coming from outside, I can understand the machine is panicking. I am going home now. I have now spent two full days on this. I have a report due today and will work on that. Christian Zimmermann FIGUGEGL! Department of Economics University of Connecticut 341 Mansfield Road, Unit 1063 Storrs, CT 06269-1063 http://ideas.repec.org/zimm/ christian.zimmermann(a)uconn.edu http://ideas.repec.org/e/pzi1.html

2 1

nebka down?
by Christian Zimmermann 15 Mar '08

15 Mar '08

I am not able to communicate with nebka. It was functional at 7:21 (confirmation mail sent to an author), and I am trying since 7:55. Any idea of something launched in between that could have brought a panic? Christian Zimmermann FIGUGEGL! Department of Economics University of Connecticut 341 Mansfield Road, Unit 1063 Storrs, CT 06269-1063 http://ideas.repec.org/zimm/ christian.zimmermann(a)uconn.edu http://ideas.repec.org/e/pzi1.html

2 5

nebka is now running
by Christian Zimmermann 15 Mar '08

15 Mar '08

Tim got it finally to work. I must have done something in the RAID configuration utility that erase the tables on sdb1. The current state of the system is: kernel 2.6, ext3 filesystem with dir_index feature, empty sdb1, boot and root on sda1. Note: Time added the dir_index feature also to sda1. This allows better handling of large filesystems, but works only with 2.6 Tim is convinced, and I agree, that we do not have a hard drive problem. The problem is software related and has to do with the fact that there is an awful lot of disk I/O going on on this machine. We should assess all the rsync's and such running and see whether they are necessary, and whether they needed at the current frequency. We should also be used the second drive to distribute the I/O load optimally across the two drives. Say, put only /home on sdb1, or only /home/aras. My sense this strategy would also be valid on raneb, snefru, etc., which seem to have disk emergencies more often than usual... Immediate measures: 1) put the RePEc Author Service back online. We were having recently 15-40 new authors a day signing up. We do not want to discourage new users. 2) Think hard how to optimize disk load 3) Then only implement new strategy. Unless there is disagreement, I put the service back on in an hour. Christian Zimmermann FIGUGEGL! Department of Economics University of Connecticut 341 Mansfield Road, Unit 1063 Storrs, CT 06269-1063 http://ideas.repec.org/zimm/ christian.zimmermann(a)uconn.edu http://ideas.repec.org/e/pzi1.html

2 1

RAS back up
by Christian Zimmermann 14 Mar '08

14 Mar '08

I have restarted the RePEc Author Service. Sune or Kit, please change the DNS record of authors.repec.org back to nebka.openlib.org or nebka.uconn.edu. Thanks! Christian Zimmermann FIGUGEGL! Department of Economics University of Connecticut 341 Mansfield Road, Unit 1063 Storrs, CT 06269-1063 http://ideas.repec.org/zimm/ christian.zimmermann(a)uconn.edu http://ideas.repec.org/e/pzi1.html

1 0

Nebka disk
by Christian Zimmermann 14 Mar '08

14 Mar '08

Nebka load is still high, so I have not done anything yet. I looked into what I am supposed to do, and I must confess I do not understand what I am supposed to do and especially what to do if something does not work right. I have attached a file containing detailed instructions and questions. Please amend so that I can work with it, hopefully tomorrow. Also, how long is this supposed to take? Christian Zimmermann FIGUGEGL! Department of Economics University of Connecticut 341 Mansfield Road, Unit 1063 Storrs, CT 06269-1063 http://ideas.repec.org/zimm/ christian.zimmermann(a)uconn.edu http://ideas.repec.org/e/pzi1.html

2 32

RAS downtime
by Christian Zimmermann 13 Mar '08

13 Mar '08

I am heading now to the server farm to do hardware maintenance on nebka. This may take a few hours. Kit or Sune, please redirect authors.repec.org to repec.econ.uconn.edu until further notice. Thanks. Christian Zimmermann FIGUGEGL! Department of Economics University of Connecticut 341 Mansfield Road, Unit 1063 Storrs, CT 06269-1063 http://ideas.repec.org/zimm/ christian.zimmermann(a)uconn.edu http://ideas.repec.org/e/pzi1.html

1 0

nebka work
by Christian Zimmermann 10 Mar '08

10 Mar '08

If you are doing any work on nebka that presents a risk of panic or shutdown, please wait until Tuesday PM (Eastern Time). I will be off campus or teaching. Christian Zimmermann FIGUGEGL! Department of Economics University of Connecticut 341 Mansfield Road, Unit 1063 Storrs, CT 06269-1063 http://ideas.repec.org/zimm/ christian.zimmermann(a)uconn.edu http://ideas.repec.org/e/pzi1.html

2 7