- ArchEc-run - RePEc Lists

Re: [ArchEc] TR: Repec: archive pdf
by Thomas Krichel 04 Dec '19

04 Dec '19

1430-FONDATION-UT(a)banque-france.fr writes > Can you tell me where you are in relation to the development of your project? I keep a diary of all the work, by the minute http://archec.repec.org/losheim This shows 34 hours of work done. However, the diary is not quite complete. It does not account for the work that I did with Jose Manuel Barrueco Cruz, when we met in October. I had to work with him to get the access to the data and understand how it is being organized. I just could not hang around and count by the minute as a usually do, even making a new entry when I go to the toilet. In general, there are many more hours of work on the project because the billable work excludes any sysadmin work, documentation and report writing. The billable work is only for actual software writing. It’s only for raw time on the computer typing the actual software. I call it parch for something like “paper archiver”. It is fairly generic, and it has to be because we have several sources of data to build the actual archive. We have data that we bring in now, and we have the historic data in the CitEc PDF storage. The main reason that there are not more hours done is as follows. When I made the application, I assumed that the project that I get paid to work on, called Cirtec---which is very vaguely related to CitEc, but must not be confused with it---would close in June. This assumption turned out to be wrong. The Cirtec project is supposed to close at the end of the year, in fact my boss had to write the final report by November 25th. When I got to know that Cirtec would not close until the end of the year happened, I hoped to be able to squeeze work in on Losheim over the time until the end of the year. Unfortunately the work on Cirtec intensified and I lost track of parch. Once I loose track of the project it takes a few days to get back into into it. I got partly back to it when I met Jose Manuel Barrueco Cruz in October. At that time I officially took an annual vacation from Cirtec. While on a meeting with him we worked on the inclusion of his CitEc archive into the RePEc parch storage. Parch is conceived to make it easy to handle data from various source. But collection the data from the CitEc source is not easy. We knew that, and that’s why the Losheim says we will attempt. In the latest run, yesterday, I only managed to find RePEc handles for 170000 of the about 800000 PDF documents that are in the storage. Clearly this needs more work. You may wonder why this is not trivial. Well the answer is that we don’t have a historic set of RePEc data available. That is, we have PDF files where we don’t have a clear indication of what RePEc paper the PDF instantiates. Building a system that would preserve the RePEc metadata is not part of Losheim. It is supposed to be part in a second application to the foundation. Clearly if the current work is not completed by January, RePEc will not apply for funding for that second phase in 2020, but only in 2021. Looking on the bright side, I have been extremely frugal in the way I count the hours. I am confident that all the required work can be done in the 100 hours that I counted. In fact the bulk of the work on parch is done. We store PDFs that we can directly find in the NEP data---which handles the current data going forward. This may sound primitive. But the system has been conceived to handle a wide variety of sources of data. This it can be configured to handle various XML-based metadata formats. It can point to various components having various functions. And it is robust to different payloads. That means it can store various instances of payloads available a URL in metadata, within the same storage file. The code is in python. That’s not a language I have used before, but we can’t use Perl, the lingua franca of RePEc because it has a steadily declining community of users and no library to handle warcs. There are 486573 warc files in storage at the last count. As for Cirtec (the project that blocks my time), I’m waiting for the November wage payment. Then I will limit my work to strictly two hours a day. With that I can finish the 100 hours by mid January. But it is not completely sure that that will do all we have set in Losheim (basically parch plus import of CitEc archive into parch data) done. Plus the work has to be documented and published. The 100 hours don’t pay for that, I’m supposed to do it for free. Equally Losheim does not provide for the data actually be used. The plan is to use it in NEP. That will only start next year. Frankly, I’d rather be paid when all the work is done. But I understand they have to close the books this year. In that case it seems reasonable to pay now, knowing there will be no further subsidy in case I can’t demonstrate during next year that I’m worthy of a 2021 subsidy. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

1 0

Re: [ArchEc] Repec: archive pdf
by Thomas Krichel 15 Jul '19

15 Jul '19

Guillaume.HORNY(a)banque-france.fr writes > As a return, we would be fine with the Foundation's logo appearing > on the webpage of the Archec project and the other types of support > refereed to into the application (links to works in which the > Foundation is involved This is done. > and advertising through NEP reporting emails). There I will need to know exactly what you want to do. At the moment all advertising opportunities are free in the sense, I have not sold anything. I wanted to try selling from in the second half of the year, provided, as I expected, that I loose my Russian job. Anyway, there are two types: issue sponsorship and report sponsorship. Issue sponsorship is for an issue, i.e. on all reports with a certain data. Report sponsorship is more static, it would appear on the report for many issues. The report sponsorship has some rather stringent formatting rules that come from the way the report issues are formatted when they are displayed in text. Issue sponsorship is a bit freer. Since I am coming late to the reply, we can extend the period for you to place adverts. The easiest would be for you to do a report sponsorship for nep-mon, the text of which we could then change to incorporate some message that you want to place. We could run this advertising until the end of 2020. I feel ashamed to come back to you on that part of your message so late. When I got the award, I was ill, I got more ill until about middle of April. By then a ton of undone work had accumulating, including two complete server migrations, with sponsors continuing to pay for two server as the new server was not ready. The timing of the migration was somewhat beyond my control, it fell badly. Thus does not excuse that I procrastinated over this. At the danger of rambling on, I expected my Russian small job to end in June, but it did not. I now more firmly think it will end by the end of the year. It pays $800 a months, roughly half my current income so it's important money from me. I spent too much time on it, running the risk that working on it would endanger me finishing the 100 hours for Losheim. At this time, I have registered 20 hours work done. This is coding only. I don't count time spent on sysadmin at all, and don't count research on how to solve a coding problem, and neither do I include time spent on examining data. Like today, I spent time looking at 32 warc files where the headers say it's PDF but can't find PDF in the payload, to determine what is wrong. I now have about 29 cases left of various sorts of data that one can make a PDF file for. I plan to store them in a particular "bedlam" folder and provide for ways for an indexer to bring them back to life, once I, or somebody else, can get them converted into PDF. Anyway the forecast is that is less than 15 hours more, I will have a basic downloader that will take proper account of multiple full-text links and multiple historic payloads. Then the task is to make some crude optimisation to run in more often with recent data, and then I can start on the tricky part, i.e., the integration of historic data. Now 20 hours may seem very little and yes, it is too little for now, but it is good to underline that a lot of sysadmin prep work had to be done. It's still time spent but not counted. Again, I profusely apologise my procrastination. The good thing is that I am procrastinating on the project itself, I feel good about what I have done. It's pretty much free of spaghetti code. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

1 0

Losheim
by Thomas Krichel 15 Jul '19

15 Jul '19

Probably most of you will know, Losheim has been approved in the Banque de France Foundation for Economic Research. (henceforth: BFFER). The proposal is linked to from the homepage http://archec.repec.org. I will copy futher corrpondance with the funders here. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

1 0

Re: [ArchEc] Working Papers, University of Southern Denmark, Centre for Border Region Studies (fwd)
by 'Christian Zimmermann' 26 Oct '17

26 Oct '17

While the papers will stay online, still a case for ArchEc. Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist ---------- Forwarded message ---------- Date: Thu, 26 Oct 2017 15:03:02 +0000 From: Olivier Walther <ow(a)sam.sdu.dk> To: 'Christian Zimmermann' <zimmermann(a)stlouisfed.org> Subject: RE: Working Papers, University of Southern Denmark, Centre for Border Region Studies Christian, Yes, the files will remain on our server. Best regards Olivier -- Olivier J. Walther, Ph.D. Associate Professor University of Southern Denmark DK-6400 Sønderborg Denmark +45.6550.8393 -----Original Message----- From: 'Christian Zimmermann' [mailto:zimmermann@stlouisfed.org] Sent: Thursday, October 26, 2017 11:00 AM To: Olivier Walther <ow(a)sam.sdu.dk> Cc: Steen Bo Frandsen <sbf(a)sam.sdu.dk> Subject: Re: Working Papers, University of Southern Denmark, Centre for Border Region Studies Are going to stay online? Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist On Thu, 26 Oct 2017, Olivier Walther wrote: > Dear Christian, > > I am writing to inform you that the Centre for Border Region Studies at the University of Southern Denmark has decided to discontinue its working paper series (https://ideas.repec.org/s/sdn/wpaper.html) Please let me know if there is anything we should do on our side. > > Best regards, > > Olivier > > -- > Olivier J. Walther, Ph.D. > Associate Professor > University of Southern Denmark > DK-6400 Sønderborg > Denmark > +45.6550.8393 > >

1 0

letter of inquiry to Sloan
by Thomas Krichel 25 Jul '16

25 Jul '16

I want to look for funding for ArchEc. The Sloan Foundation encourage letters of inquiry. Instructions are here http://www.sloan.org/apply-for-grants/letters-of-inquiry/ Here is my draft http://openlib.org/home/krichel/tmp/archec.txt There are two main points that I think I want to discuss. 1 - I am not sure the OLS should be the grantee. 2 - I am not sure we should / need to go through the board. The board has never been used and getting it to work will take a bit of time and effort. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

3 6

access function
by Thomas Krichel 24 Jul '16

24 Jul '16

I placed my function to access the store at http://ernad.openlib.org/perl/Ernad/FullTextFetch.pm This is not really where it should be but for the moment that location will do. To fix terminology, I suggest we will address the store as "pafis". It's an acronym (?) for something like "paper files store". -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

1 0

Re: [ArchEc] Quick, for ArchEc
by Thomas Krichel 24 Feb '14

24 Feb '14

'Christian Zimmermann' writes > RePEc:deg is closing. I will continue to host the metadata, but you > may want to grab the pdfs before they disappear. Thank you. I have done that! -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

1 0

template archive
by Thomas Krichel 18 Sep '13

18 Sep '13

I'm pleased to report some progress on templating component of ArchEc. I am collecting templates now from the tarballs I have. The script is at http://archec.repec.org/bin/run_archec The archive is at http://archec.repec.org/archive -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

1 0

software gone?
by Thomas Krichel 11 Sep '13

11 Sep '13

I am looking at ArchEc againn. Did Rishab remove the software that he wrote? In .bash_history, I find rm -rf bin rm -rf bin.tar.gz rm -rf archive/ rm -rf handle* rm -rf count_* rm -rf t* rm -rf create_checksum rm -rf read_redif_ rm -rf read_redif_* rm -rf saveURL rm -rf ReDIF_count.pm -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

1 0

software review
by Thomas Krichel 31 Mar '13

31 Mar '13

Rishabh Sharma writes > Repec-2009-09-13.tar.gz > > changed_templates:: 187172 > > unchanged_templates:: 727492 > > NEW TEMPLATES:: 13473 > > LOST TEMPLATES:: 1376 what excutble is used to do this? if it is ~/bin/read_redif, then it looks completely wrong. In archec@modin:~/lib/perl/ArchEc/ReDIF.pm, I read if ( -e $tobi_path) { print "File exist"; $unchanged_templates++; print "\n\n\n"; return; } If a template was prevously saved, at unchange tobi time it is considered to be unchanged? And then else{ print "\n"; print "creating Directory ".$ENV{"HOME"}."/archive/".$tobi_dir; make_path($ENV{"HOME"}."/archive/".$tobi_dir); print "\n"; print "creating tobi file ".$tobi_path; open FILE,">$tobi_path" or die $!; print FILE $template->{'TEXT'}; #binmode(FILE); #open FILE1,">$tobi_path.md5" or die $!; # print FILE1 Digest::MD5->new->addfile(*FILE)->hexdigest." $$tobi_filename\ "; close FILE; $changed_templates ++; print "\n\n\n"; } Meanning when the file time of ReDIF file changes, a new tobi file is written? This will needlessly duplicate contents. It looks like there is no serious attempt to distinguish between changed and unchanged templates by checksum. All tobis have to go through this check, irrespective of origin. Cheers, Thomas Krichel http://openlib.org/home/krichel http://authorprofile.org/pkr1 skype: thomaskrichel

1 0