I have just opened the vault and the plind for your rsyncing pleasure krichel@trabbi/tmp$ mkdir plind krichel@trabbi/tmp$ rsync -av rsync://archec.repec.org/plind/ plind/ The plind is the payload index. It says where in the vault file PDF data for papers can be found. The start of the payload is at 'b', the length at 'f'. The 'o' field has the PDF status. 'm' according to mime type 'a' it has something "%PDF" inside first 100 bytes 'p' it has "PDF" in the futli, important for ftp 'f' it has an URL starting with "ftp://" 'r' is from a WARC resource record that contains a payload, i.e. not preceeded by a WARC metadata record, or not concurrent to another record. At this time a tiny fraction of the plind data is available. I'm still running a full set. The vault contains the actual warcs. I recommend to exclude the cdx files krichel@trabbi/tmp$ mkdir vault krichel@trabbi/tmp$ rsync -av --exclude '*.cdx' rsync://archec.repec.org/vault/ vault/ At this time, there is no way to actually limit this to files that are actually mentioned in the plind. This is important since we hold PDF only for a minority of papers. Suggestions welcome. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel