- CollEc-run - RePEc Lists

paths under management
by Thomas Krichel 07 Dec '14

07 Dec '14

I checked today. We now have more than one billion paths under management icanis@katri:~/perl$ ./count_paths total is 1022613377 -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

1 0

Re: [CollEc] CollEc
by Thomas Krichel 21 Nov '14

21 Nov '14

Nikos Askitas writes > I hope that you remember me I have a very poor memory, but I remember. > I have been looking at CollEc recently and started programming the > search interface a bit. CollEc has a search interface. > I would basically like to write a script which will get the shortest > path(s) between two authors. So I would like to say > get_paths(Askitas, Krichel) and get the data in some form I can work > on further. I can strip the HTML with regexs I'd rather give you the complete dataset. > 1. Is there a file which maps RealName to repec handles? Can you point me somewhere? icanis@katri:~/icanis/input$ cat ras_mans_nodes_1416097129.xml | head -10 <nodes> <node ref="paa1" name="Arild Aakvik" homepage="http://people.uib.no/secaa"/> <node ref="paa11" name="Arnstein Aassve" homepage="http://faculty.unibocconi.eu/arnsteinaassve/"/> <node ref="paa12" name="susan aaronson" homepage="http://www.gwu.edu/~elliott/faculty/aaronson.cfm"/> <node ref="paa13" name="Daniel Aaronson"/> <node ref="paa2" name="David Aadland" homepage="http://www.uwyo.edu/aadland/"/> <node ref="paa22" name="Knut Are Aastveit" homepage="http://www.norges-bank.no/en/about/research/economists/aastveit-knut-are/"/> <node ref="paa23" name="Knut Aase" homepage="http://www.nhh.no/Default.aspx?ID=2004"/> <node ref="paa6" name="Rolf Aaberge"/> <node ref="paa8" name="Rob Aalbers"/> > 2. Do you have a script somewhere which would work for pairs of names? No, but I have the data. Let's look at at example icanis@katri:~/icanis/input$ grep Krichel ras_mans_nodes_1416097129.xml <node ref="pkr1" name="Thomas Krichel" homepage="http://openlib.org/home/krichel"/> icanis@katri:~/icanis/input$ grep Askitas ras_mans_nodes_1416097129.xml <node ref="pas55" name="Nikos Askitas" homepage="http://www.iza.org/home/askitas"/> This gives us the handle, from there we can find the paths in two ways. icanis@katri:~$ grep 'pas55$' icanis/paths/ras/biwe/k/r/1/paths 5 pzi1 pur18 psc166 pzi13 pas55 icanis@katri:~$ grep 'pkr1$' icanis/paths/ras/biwe/a/s/55/paths 5 pzi13 psc166 pur18 pzi1 pkr1 I use the latest of the two path files that I can grep. That's how the search works. BTW, there are close to 1 billion paths. > The search cgi you have now works for Name and handle or for two > handles. Yes, I should write an interface for two. > 3. Is there a cgi you can post which will deliver shortest paths as > arrays of arrays without html? Even a path per line separated by > comma would do. I have not done this, but you could do it. > 4. Would you share ALL the coauthorship data with me? Sure. Currently I don't have a backup of the paths, since they can be recomputed. But it would take a very long time. If you would provide a backup for me that would help me. Kindly open an account "icanis" on a machine with rsync installed and authorise the key icanis@katri:~$ cat ~/.ssh/id_rsa.pub ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOhbsMyNTv9O3h0tu7eMjqdR5t2BG1jxqErz2YVoVcPGfSZ2mOgwMLMSeFQY3YwNzkC/lfzbtGKybm9dl+CfvnERKL3tA5+NiQiXyJ0WZVp/bEvePMMsDzcLv+Z4RHStl9mU+9OuTPDYzpheTWbjswszQc9kE2ECqlRUqfIb1MQ1/UWDfaq9p8O0USYc25t/uM95VOtHTPHvy75d2+8m3NJPhdFX2yqTiD0ra2phdwMrgw+Csivgs/gfhIhMepep29+V+sBvHDhuVOVBS9N87ae7AJlw5HDceIKwkxasFfMB9mw7RxK7OGTGBcM78ykFRustlKAiDy3M9t/iqb9NNh icanis@fricka This is the size of the account right now icanis@katri:~$ du -s . 81919236 . > I would like to write some papers with it? Write as many or as few as you like. Three additional points. (1) If you can calculate summary statistics on the network, I can use them in the web site. The site is written using XSLT, so an XML file with the stats will be perfect. (2) I have taken liberty to add you to the Collec-run mailing list. You are the sixth member on the list. It has very little traffic, but all correspondance about CollEc should be sent there. (3) I was in Cologne at GESIS between October 13 and 31. I hiked even do the Rheinsteig between Bonn and the Drachenfels on 25th October so I could have met you. But I had forgotten about you, sorry. I am back in NYC now, homeless and jobless, so if you have a consultancy opportunity I would look into it. I will probably be back at GESIS at some stage. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

1 0

Re: [CollEc] repec genealogy/citation/coauthor raw data
by Lars Vilhuber 08 Nov '14

08 Nov '14

P. S. What lead you to pick (single or sometimes dual) advisor, and not allowing for committee members? I've heard it a few times from folks I showed the genealogy page. Any plans to add that capability? -- Sent from my mobile device On Nov 7, 2014 2:02 PM, 'Christian Zimmermann' <zimmermann(a)stlouisfed.org> wrote: Hallo lars, gaebe es die Tage einen guten Zeitpunkt zu telefonieren? Eine ganz praktische Frage. Wir finden in unseren OH QCEW daten relativ viele Firmen, die nur einen quarter existieren (~20%). Denkst du das ist eine realistische Groessenordnung? Viele Gruesse, Till

2 1

Re: [CollEc] repec genealogy/citation/coauthor raw data
by Lars Vilhuber 07 Nov '14

07 Nov '14

Then send me a mysqldump. That'll do just fine. -- Sent from my mobile device On Nov 7, 2014 2:02 PM, 'Christian Zimmermann' <zimmermann(a)stlouisfed.org> wrote: Tell me in what format you want the genealogy data. I have it in MySQL. Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA http://ideas.repec.org/zimm/ @CZimm_economist On Fri, 7 Nov 2014, Lars Vilhuber wrote: > Christian, Thomas: > > just a brief update. We've identified a (very) qualified (undergrad) RA, who'll work on this part time starting mid-January. > > We'll start with a demo using a (static) data dump (I have grabbed a one-time copy of Thomas' data, will still need a dump of the genealogy data, unless the RA simply decides to scrape it), we can discuss how to most efficiently "collect" the data necessary for graphing and exploring in an "implemented production" server, once we get there. > > I'm excited! > > Lars > > -- > Lars Vilhuber | Executive Director > Economist | Labor Dynamics Institute > lars.vilhuber(a)cornell.edu -+- Cornell University > Office/Cell : +1.607-330-5743 | ILR School - Department of Economics > Fax : +1.866-873-9078 + U.S. Census Bureau - CES - LEHD > > Labor Dynamics Institute: http://www.ilr.cornell.edu/ldi > Visit the Cornell VirtualRDC: http://www.vrdc.cornell.edu > > ________________________________________ > From: 'Christian Zimmermann' <zimmermann(a)stlouisfed.org> > Sent: Tuesday, October 28, 2014 11:03 > To: Lars Vilhuber > Cc: Thomas Krichel; CollEc Run > Subject: Re: repec genealogy/citation/coauthor raw data > > I do not (yet) have a way to share the geneaology data, but we can talk > about what would best suitable. > > My optimal outcome would to have an app that can be used on CollEc, your > site and the RePEc Genealogy to show the respective networks. CollEc and > RG would be limited to their data, you could show all you want. > > Christian Zimmermann FIGUGEGL! > Economic Research > Federal Reserve Bank of St. Louis > P.O. Box 442 > St. Louis MO 63166-0442 USA > http://ideas.repec.org/zimm/ @CZimm_economist > > On Tue, 28 Oct 2014, Lars Vilhuber wrote: > >> Thomas, Christian: >> >> (I'll continue the discussion in English, since I might want to forward this to my RAs) >> >> This is doable! >> >> I may have a very competent undergraduate RA to work on this in the Spring. I'll have more details (on his availability) in about 2 weeks. >> >> I'm putting together a quick write-up of why this makes sense from our projects, and why it might make sense for RePEc (or in general). I'm aiming for something that would be maintainable as an actual web app on (your? our?) servers as part of the network. >> >> I can see all the data (I think) I need on the page that Christian sent me, except for the genealogy data. Possibly Thomas has data in a more concise format than us parsing the RePEc:per archive? But that's no problem. Can you provide me with a one-time dump (extract probably is NOT sufficient if done randomly, but could be time-based - birthday of the person in the database?) for now of the genealogy database? (what format are you using there? ) >> -- >> Lars Vilhuber | Executive Director >> Economist | Labor Dynamics Institute >> lars.vilhuber(a)cornell.edu -+- Cornell University >> Office/Cell : +1.607-330-5743 | ILR School - Department of Economics >> Fax : +1.866-873-9078 + U.S. Census Bureau - CES - LEHD >> >> Labor Dynamics Institute: http://www.ilr.cornell.edu/ldi >> Visit the Cornell VirtualRDC: http://www.vrdc.cornell.edu >> >> ________________________________________ >> From: Thomas Krichel <krichel(a)openlib.org> >> Sent: Saturday, October 25, 2014 03:17 >> To: Lars Vilhuber >> Cc: CollEc Run; Christian Zimmermann >> Subject: Re: repec genealogy/citation/coauthor raw data >> >> Lars Vilhuber wrote: >> >>> this is what I was aiming to do: >>> >>> http://www.vrdc.cornell.edu/repecgraph/ >>> >>> in my abundant free time. I have a programmer who would be perfectly >>> capable of doing that, but no time right now (may change in a >>> year). Also, if you are interested, I could propose this to one of >>> our Cornell CS classes - they do 'client-oriented visualization >>> projects' and could probably do this with a dump of the data, if not >>> an API. >>> >>> This is just idle thoughts, but who knows... >> >> 'Christian Zimmermann' writes >> >>> I always wanted http://collec.repec.org/ to have something like this. >> >> Me to. But I don't have the expertise. >> >>> Thomas may help you getting access to the CollEc data/server for >>> starters. >> >> I surely will be pleased to do that. >> >> >> -- >> >> Cheers, >> >> Thomas Krichel http://openlib.org/home/krichel >> skype:thomaskrichel >> >

1 0

Re: [CollEc] repec genealogy/citation/coauthor raw data
by Thomas Krichel 07 Nov '14

07 Nov '14

Lars Vilhuber wrote: > this is what I was aiming to do: > > http://www.vrdc.cornell.edu/repecgraph/ > > in my abundant free time. I have a programmer who would be perfectly > capable of doing that, but no time right now (may change in a > year). Also, if you are interested, I could propose this to one of > our Cornell CS classes - they do 'client-oriented visualization > projects' and could probably do this with a dump of the data, if not > an API. > > This is just idle thoughts, but who knows... 'Christian Zimmermann' writes > I always wanted http://collec.repec.org/ to have something like this. Me to. But I don't have the expertise. > Thomas may help you getting access to the CollEc data/server for > starters. I surely will be pleased to do that. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

3 7

Re: [CollEc] repec genealogy/citation/coauthor raw data
by Thomas Krichel 28 Oct '14

28 Oct '14

Lars Vilhuber writes > I'm in. What should I look for? icanis@katri:~$ cd icanis/ icanis@katri:~/icanis$ ls cgi-bin doc html input log opt paths perl var xsl icanis@katri:~/icanis$ ls input ras_mans_edges_1414023465.xml ras_mans_nodes_1414023465.xml ras_mans_texts_1414023465.xml ras_mans_edges_1414109858.xml ras_mans_nodes_1414109858.xml ras_mans_texts_1414109858.xml ras_mans_edges_1414196238.xml ras_mans_nodes_1414196238.xml ras_mans_texts_1414196238.xml ras_mans_edges_1414282648.xml ras_mans_nodes_1414282648.xml ras_mans_texts_1414282648.xml ras_mans_edges_1414369135.xml ras_mans_nodes_1414369135.xml ras_mans_texts_1414369135.xml ras_mans_edges_1414455515.xml ras_mans_nodes_1414455515.xml ras_mans_texts_1414455515.xml Input data, copies for various time stamps. icanis@katri:~/icanis/paths/ras/biwe$ Paths files icanis@katri:~/icanis/paths/ras/biwe/k/r/1$ head paths 5 ppe279 pgi22 pna69 ppe165 psy16 5 pzi1 psc208 phe187 pmi41 pje110 7 ple338 pba495 pro56 pmi460 ppr70 pca142 pam43 5 ppe279 psa83 pfe14 pga60 pma1751 6 pzi1 pur18 pwa567 pto21 pho90 pbo652 6 pse31 pro263 pku26 pvi52 pro647 ppo316 6 ppe279 pba671 pmc43 plu143 pbu323 pce142 5 ple338 pzi83 pgi131 pya75 pma1285 7 ple338 pba203 pni25 pva45 piw9 pya176 pra722 7 ppe279 psa83 pma173 pad1 pbi75 pka372 ple294 length ... path. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

1 0

Re: [CollEc] CollEc used in research
by Thomas Krichel 09 May '14

09 May '14

'Christian Zimmermann' writes > https://twitter.com/vndimitrova/status/464766621186142208/photo/1 > > I cannot find the paper, though. Thank you. BTW, I have set up processes on modin to help with the calculations on katri. This works via NFS mount. Thus I'm freening resources on katri and raising the amount of processes running to do the calculations from 8 to 10. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

1 0

Re: [CollEc] CollEc data request
by Thomas Krichel 11 Mar '14

11 Mar '14

Richard Tol writes > I wondered whether you would be able to give me access to the > collaboration data of CollEc? I have just scraped the RePEc > genealogy data, which took 8 days. CollEc would probably take a > month or more. Send me a user name you want to have and an ssh public key. I will give you access to the box where the data is being held. Then we can arrange for a call to discuss. > More importantly, did you archive earlier CollEc data? One would need snapshots of RAS data. I have some of these. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

1 0

Re: [CollEc] CollEc question
by Thomas Krichel 06 Sep '13

06 Sep '13

Spencer Banzhaf writes > Thank you for your work at REPEC and CollEc. In recently exploring > CollEc, I noticed that a lot of my co-authors do not appear in my > official list, even some who are registered and with large networks. Holy cow! That's sounds very serious. > For example, consider this paper: > http://logec.repec.org/scripts/paperstat.pf?h=repec:wly:jpamgt:v:29:y:2010:… > > The paper is in REPEC and in my profile, my coauthors Oates and Sanchirico are registered in REPEC, yet they do not appear in my co-author list. > Upon investigation, this paper appear to be neither in Oates profile http://ideas.repec.org/e/poa3.html nor in Sarchirico's profile http://ideas.repec.org/e/psa58.html > Is this a bug? I don't have evidence yet. For any paper that would establish a link, your co-authors have to have claimed it. This was part of the reasons for doing CollEc: get authors to ask their co-authors to register and maintain their profiles. > Is there some way to fix it? Yes, contact your co-authors to ask them to add the paper to their profiles. Note that CollEc updates slowly because of the huge amount of calculations required; but if you get a new connection you should see your ranking increase over the space of about a month. Yes, it takes that long. Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel

1 0

removal of author links
by 'Christian Zimmermann' 08 Jun '13

08 Jun '13

Does CollEc handle the removal of links between authors? The reason I ask is that the Journal of Economic Literature was previously indexing all authors of book reviews within the same issue as co-authors. I got them to stop that, but some authors have in the meanwhile claimed articles that are now being dropped. Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA http://ideas.repec.org/zimm/

2 5