'Christian Zimmermann' writes
2) I notice today the ReDIF-perl was not installed on RAS. after installation though, emails have broken. the to field is malformed, for example:
The following message to <"Christian Zimmermann" <zimmermann@stlouisfed.org>@mx1.stlouisfed.org> was undeliverable.
returning to 2.70 fixed the problem, so it has nothing to do with any external perl module that refreshed during install. I suspect this has to do with the following in Parser.pm:
< 'remove_newline_from_values' => 0,
#SK 'remove_newline_from_values' => 0, Obsoleted by new Unicode processing, newlines are always removed
or at least that is all I can see.
I have spent several hours on this, need some sleep now and am open for suggestions or inspiration. RAS code is horrible.
Tell me about it. ;-). If you give me access I will try to fix. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
2) I notice today the ReDIF-perl was not installed on RAS. after installation though, emails have broken. the to field is malformed, for example:
The following message to <"Christian Zimmermann" <zimmermann@stlouisfed.org>@mx1.stlouisfed.org> was undeliverable.
returning to 2.70 fixed the problem, so it has nothing to do with any external perl module that refreshed during install. I suspect this has to do with the following in Parser.pm:
< 'remove_newline_from_values' => 0,
#SK 'remove_newline_from_values' => 0, Obsoleted by new Unicode processing, newlines are always removed
or at least that is all I can see.
There are many differences between 2.70 and 2.74. It's not clear to me that they should cause problems with sending e-mails. At least I have not experienced any problems. Which process is sending the e-mail, where does it get the e-mail address? Perl module for sending e-mail or system call to an external process? I suppose an assumption of a newline (or white space) at the end of the e-mail address that no longer holds could cause problems when building a command line. /Sune
Sune Karlsson writes
Which process is sending the e-mail,
I suspect that these are ARPU mails. ARPU itself is very complicated. It's kind of you to try but without a actual system at hand nobody but Ivan is likely to find where the error is. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
The latest of the little I was able to figure out. ReDIF-perl does not seem to be to blame, at least not directly. Updating it triggered all sort of other updates, and despite reverting ReDIF-perl, emails are *not* going out. New registrants and email changes bounced hopping within our system. Emails are typically formatted like zimmermann@stlouisfed.org@mx1.stlouisfed.org but I have seen one split in two like this: christian.zimmermann@stls.frb.o@mx1.stlouisfed.org rg@mx1.stlouisfed.org which makes absolutely no sense. There must be some package that has started messing with those email addresses. From the list of updated packages, there is no obvious candidate. Still digging. And, no, Thomas I cannot give you access to the machine. Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist On Sun, 10 Apr 2016, Sune Karlsson wrote:
2) I notice today the ReDIF-perl was not installed on RAS. after installation though, emails have broken. the to field is malformed, for example:
The following message to <"Christian Zimmermann" <zimmermann@stlouisfed.org>@mx1.stlouisfed.org> was undeliverable.
returning to 2.70 fixed the problem, so it has nothing to do with any external perl module that refreshed during install. I suspect this has to do with the following in Parser.pm:
< 'remove_newline_from_values' => 0,
#SK 'remove_newline_from_values' => 0, Obsoleted by new Unicode processing, newlines are always removed
or at least that is all I can see.
There are many differences between 2.70 and 2.74. It's not clear to me that they should cause problems with sending e-mails. At least I have not experienced any problems.
Which process is sending the e-mail, where does it get the e-mail address? Perl module for sending e-mail or system call to an external process? I suppose an assumption of a newline (or white space) at the end of the e-mail address that no longer holds could cause problems when building a command line.
/Sune
Some thoughts off the top of my head. Most probably random noise or things you have already thought of. Is mx1.stlouisfed.org your mail gateway? Is this configured somewhere or picked up automatically? If configured it might help to trace where this variable is used. Perl module or system call for sending e-mail? I doubt that something is "messing" with the e-mail address. It is more likely that whatever is sending the e-mail tacks on the @mx1.stlouisfed.org to the address for some strange reason. /Sune
-----Original Message----- From: 'Christian Zimmermann' [mailto:zimmermann@stlouisfed.org] Sent: Sunday, April 10, 2016 8:03 PM To: Sune Karlsson Cc: Thomas Krichel; RAS-run Mailing List; Hayes, Dan -- Dan Hayes Subject: RE: [RAS] ReDIF-perl trouble
The latest of the little I was able to figure out.
ReDIF-perl does not seem to be to blame, at least not directly. Updating it triggered all sort of other updates, and despite reverting ReDIF- perl, emails are *not* going out. New registrants and email changes bounced hopping within our system.
Emails are typically formatted like zimmermann@stlouisfed.org@mx1.stlouisfed.org
but I have seen one split in two like this: christian.zimmermann@stls.frb.o@mx1.stlouisfed.org rg@mx1.stlouisfed.org
which makes absolutely no sense.
There must be some package that has started messing with those email addresses. From the list of updated packages, there is no obvious candidate.
Still digging.
And, no, Thomas I cannot give you access to the machine.
Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://mailfilter.sunet.se/canit/urlproxy.php?_q=aHR0cHM6Ly9pZGVhcy5yZX BlYy5vcmcvemltbS8%3D&_r=b3J1LXNl&_s=c2tuQG9ydS5zZQ%3D%3D @CZimm_economist
On Sun, 10 Apr 2016, Sune Karlsson wrote:
2) I notice today the ReDIF-perl was not installed on RAS. after installation though, emails have broken. the to field is malformed, for example:
The following message to <"Christian Zimmermann" <zimmermann@stlouisfed.org>@mx1.stlouisfed.org> was undeliverable.
returning to 2.70 fixed the problem, so it has nothing to do with any external perl module that refreshed during install. I suspect this has to do with the following in Parser.pm:
< 'remove_newline_from_values' => 0,
#SK 'remove_newline_from_values' => 0, Obsoleted by new Unicode processing, newlines are always removed
or at least that is all I can see.
There are many differences between 2.70 and 2.74. It's not clear to me that they should cause problems with sending e-mails. At least I have not experienced any problems.
Which process is sending the e-mail, where does it get the e-mail address? Perl module for sending e-mail or system call to an external process? I suppose an assumption of a newline (or white space) at the end of the e-mail address that no longer holds could cause problems when building a command line.
/Sune
the mail gateway "mx1.stlouisfed.org" is configured in sendmail and postfix config files, /etc/mail/sendmail.mc and /etc/postfix/main.cf. I'm not sure where it might be included in perl code or why it would be. On Sun, Apr 10, 2016 at 2:47 PM, Sune Karlsson <Sune.Karlsson@oru.se> wrote:
Some thoughts off the top of my head. Most probably random noise or things you have already thought of.
Is mx1.stlouisfed.org your mail gateway? Is this configured somewhere or picked up automatically? If configured it might help to trace where this variable is used.
Perl module or system call for sending e-mail? I doubt that something is "messing" with the e-mail address. It is more likely that whatever is sending the e-mail tacks on the @mx1.stlouisfed.org to the address for some strange reason.
/Sune
-----Original Message----- From: 'Christian Zimmermann' [mailto:zimmermann@stlouisfed.org] Sent: Sunday, April 10, 2016 8:03 PM To: Sune Karlsson Cc: Thomas Krichel; RAS-run Mailing List; Hayes, Dan -- Dan Hayes Subject: RE: [RAS] ReDIF-perl trouble
The latest of the little I was able to figure out.
ReDIF-perl does not seem to be to blame, at least not directly. Updating it triggered all sort of other updates, and despite reverting ReDIF- perl, emails are *not* going out. New registrants and email changes bounced hopping within our system.
Emails are typically formatted like zimmermann@stlouisfed.org@mx1.stlouisfed.org
but I have seen one split in two like this: christian.zimmermann@stls.frb.o@mx1.stlouisfed.org rg@mx1.stlouisfed.org
which makes absolutely no sense.
There must be some package that has started messing with those email addresses. From the list of updated packages, there is no obvious candidate.
Still digging.
And, no, Thomas I cannot give you access to the machine.
Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://mailfilter.sunet.se/canit/urlproxy.php?_q=aHR0cHM6Ly9pZGVhcy5yZX BlYy5vcmcvemltbS8%3D&_r=b3J1LXNl&_s=c2tuQG9ydS5zZQ%3D%3D @CZimm_economist
On Sun, 10 Apr 2016, Sune Karlsson wrote:
2) I notice today the ReDIF-perl was not installed on RAS. after installation though, emails have broken. the to field is malformed, for example:
The following message to <"Christian Zimmermann" <zimmermann@stlouisfed.org>@mx1.stlouisfed.org> was undeliverable.
returning to 2.70 fixed the problem, so it has nothing to do with any external perl module that refreshed during install. I suspect this has to do with the following in Parser.pm:
< 'remove_newline_from_values' => 0,
#SK 'remove_newline_from_values' => 0, Obsoleted by new Unicode processing, newlines are always removed
or at least that is all I can see.
There are many differences between 2.70 and 2.74. It's not clear to me that they should cause problems with sending e-mails. At least I have not experienced any problems.
Which process is sending the e-mail, where does it get the e-mail address? Perl module for sending e-mail or system call to an external process? I suppose an assumption of a newline (or white space) at the end of the e-mail address that no longer holds could cause problems when building a command line.
/Sune
The configuration file for ACIS (the software for RAS) has: sendmail = '/usr/sbin/sendmail -t -f authors@repec.org' So it is not using a perl module to send email. It pipes to sendmail: my $sendmail = $config -> {sendmail}; if ( open MESSAGE, "|-:utf8", $sendmail ) { print MESSAGE $header, "\n", $body; close MESSAGE; } I am currently playing with a line that has the following comment: my $val = encode( 'MIME-Q', $value ); # a nasty hack to fix Encode's wrapping 'feature': Could be promising. It is applied to message headers. Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist On Sun, 10 Apr 2016, Dan Hayes wrote:
the mail gateway "mx1.stlouisfed.org" is configured in sendmail and postfix config files, /etc/mail/sendmail.mc and /etc/postfix/main.cf. I'm not sure where it might be included in perl code or why it would be.
On Sun, Apr 10, 2016 at 2:47 PM, Sune Karlsson <Sune.Karlsson@oru.se> wrote:
Some thoughts off the top of my head. Most probably random noise or things you have already thought of.
Is mx1.stlouisfed.org your mail gateway? Is this configured somewhere or picked up automatically? If configured it might help to trace where this variable is used.
Perl module or system call for sending e-mail? I doubt that something is "messing" with the e-mail address. It is more likely that whatever is sending the e-mail tacks on the @mx1.stlouisfed.org to the address for some strange reason.
/Sune
-----Original Message----- From: 'Christian Zimmermann' [mailto:zimmermann@stlouisfed.org] Sent: Sunday, April 10, 2016 8:03 PM To: Sune Karlsson Cc: Thomas Krichel; RAS-run Mailing List; Hayes, Dan -- Dan Hayes Subject: RE: [RAS] ReDIF-perl trouble
The latest of the little I was able to figure out.
ReDIF-perl does not seem to be to blame, at least not directly. Updating it triggered all sort of other updates, and despite reverting ReDIF- perl, emails are *not* going out. New registrants and email changes bounced hopping within our system.
Emails are typically formatted like zimmermann@stlouisfed.org@mx1.stlouisfed.org
but I have seen one split in two like this: christian.zimmermann@stls.frb.o@mx1.stlouisfed.org rg@mx1.stlouisfed.org
which makes absolutely no sense.
There must be some package that has started messing with those email addresses. From the list of updated packages, there is no obvious candidate.
Still digging.
And, no, Thomas I cannot give you access to the machine.
Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://mailfilter.sunet.se/canit/urlproxy.php?_q=aHR0cHM6Ly9pZGVhcy5yZX BlYy5vcmcvemltbS8%3D&_r=b3J1LXNl&_s=c2tuQG9ydS5zZQ%3D%3D @CZimm_economist
On Sun, 10 Apr 2016, Sune Karlsson wrote:
2) I notice today the ReDIF-perl was not installed on RAS. after installation though, emails have broken. the to field is malformed, for example:
The following message to <"Christian Zimmermann" <zimmermann@stlouisfed.org>@mx1.stlouisfed.org> was undeliverable.
returning to 2.70 fixed the problem, so it has nothing to do with any external perl module that refreshed during install. I suspect this has to do with the following in Parser.pm:
< 'remove_newline_from_values' => 0, > #SK 'remove_newline_from_values' => 0, Obsoleted by new Unicode processing, newlines are always removed
or at least that is all I can see.
There are many differences between 2.70 and 2.74. It's not clear to me that they should cause problems with sending e-mails. At least I have not experienced any problems.
Which process is sending the e-mail, where does it get the e-mail address? Perl module for sending e-mail or system call to an external process? I suppose an assumption of a newline (or white space) at the end of the e-mail address that no longer holds could cause problems when building a command line.
/Sune
One thing that could be causing problems is if $header doesn't end with a newline. Then there would be no blank line between the headers and the body of the message. I doubt that encode would do something really strange. Have you tried printing to a file instead of piping? /Sune
-----Original Message----- From: 'Christian Zimmermann' [mailto:zimmermann@stlouisfed.org] Sent: Sunday, April 10, 2016 11:52 PM To: Dan Hayes Cc: Sune Karlsson; Thomas Krichel; RAS-run Mailing List Subject: Re: [RAS] ReDIF-perl trouble
The configuration file for ACIS (the software for RAS) has:
sendmail = '/usr/sbin/sendmail -t -f authors@repec.org'
So it is not using a perl module to send email. It pipes to sendmail:
my $sendmail = $config -> {sendmail}; if ( open MESSAGE, "|-:utf8", $sendmail ) { print MESSAGE $header, "\n", $body; close MESSAGE; }
I am currently playing with a line that has the following comment: my $val = encode( 'MIME-Q', $value ); # a nasty hack to fix Encode's wrapping 'feature':
Could be promising. It is applied to message headers.
Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://mailfilter.sunet.se/canit/urlproxy.php?_q=aHR0cHM6Ly9pZGVhcy5yZX BlYy5vcmcvemltbS8%3D&_r=b3J1LXNl&_s=c2tuQG9ydS5zZQ%3D%3D @CZimm_economist
On Sun, 10 Apr 2016, Dan Hayes wrote:
the mail gateway "mx1.stlouisfed.org" is configured in sendmail and postfix config files, /etc/mail/sendmail.mc and /etc/postfix/main.cf. I'm not sure where it might be included in perl code or why it would be.
On Sun, Apr 10, 2016 at 2:47 PM, Sune Karlsson <Sune.Karlsson@oru.se> wrote:
Some thoughts off the top of my head. Most probably random noise or things you have already thought of.
Is mx1.stlouisfed.org your mail gateway? Is this configured somewhere or picked up automatically? If configured it might help to trace where this variable is used.
Perl module or system call for sending e-mail? I doubt that something is "messing" with the e-mail address. It is more likely that whatever is sending the e-mail tacks on the @mx1.stlouisfed.org to the address for some strange reason.
/Sune
-----Original Message----- From: 'Christian Zimmermann' [mailto:zimmermann@stlouisfed.org] Sent: Sunday, April 10, 2016 8:03 PM To: Sune Karlsson Cc: Thomas Krichel; RAS-run Mailing List; Hayes, Dan -- Dan Hayes Subject: RE: [RAS] ReDIF-perl trouble
The latest of the little I was able to figure out.
ReDIF-perl does not seem to be to blame, at least not directly. Updating it triggered all sort of other updates, and despite reverting ReDIF- perl, emails are *not* going out. New registrants and email changes bounced hopping within our system.
Emails are typically formatted like zimmermann@stlouisfed.org@mx1.stlouisfed.org
but I have seen one split in two like this: christian.zimmermann@stls.frb.o@mx1.stlouisfed.org rg@mx1.stlouisfed.org
which makes absolutely no sense.
There must be some package that has started messing with those email addresses. From the list of updated packages, there is no obvious candidate.
Still digging.
And, no, Thomas I cannot give you access to the machine.
Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA
https://mailfilter.sunet.se/canit/urlproxy.php?_q=aHR0cHM6Ly9pZGVhcy5yZX
BlYy5vcmcvemltbS8%3D&_r=b3J1LXNl&_s=c2tuQG9ydS5zZQ%3D%3D @CZimm_economist
On Sun, 10 Apr 2016, Sune Karlsson wrote:
> 2) I notice today the ReDIF-perl was not installed on RAS. after > installation though, emails have broken. the to field is malformed, for > example: > > The following message to <"Christian Zimmermann" > <zimmermann@stlouisfed.org>@mx1.stlouisfed.org> was > undeliverable. > > returning to 2.70 fixed the problem, so it has nothing to do with any > external perl module that refreshed during install. I suspect this has to do > with the following in Parser.pm: > > < 'remove_newline_from_values' => 0, >> #SK 'remove_newline_from_values' => 0, Obsoleted by new > Unicode processing, newlines are always removed > > or at least that is all I can see.
There are many differences between 2.70 and 2.74. It's not clear to me that they should cause problems with sending e-mails. At least I have not experienced any problems.
Which process is sending the e-mail, where does it get the e-mail address? Perl module for sending e-mail or system call to an external process? I suppose an assumption of a newline (or white space) at the end of the e-mail address that no longer holds could cause problems when building a command line.
/Sune
There is print MESSAGE $header, "\n", $body; I have not yet been successful in printing to file. I have to wait for each email to reach max hops to see a result... Not the most elegant and quite frustrating. I am attaching what I think is the critical file, if that helps. Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist On Sun, 10 Apr 2016, Sune Karlsson wrote:
One thing that could be causing problems is if $header doesn't end with a newline. Then there would be no blank line between the headers and the body of the message.
I doubt that encode would do something really strange. Have you tried printing to a file instead of piping?
/Sune
-----Original Message----- From: 'Christian Zimmermann' [mailto:zimmermann@stlouisfed.org] Sent: Sunday, April 10, 2016 11:52 PM To: Dan Hayes Cc: Sune Karlsson; Thomas Krichel; RAS-run Mailing List Subject: Re: [RAS] ReDIF-perl trouble
The configuration file for ACIS (the software for RAS) has:
sendmail = '/usr/sbin/sendmail -t -f authors@repec.org'
So it is not using a perl module to send email. It pipes to sendmail:
my $sendmail = $config -> {sendmail}; if ( open MESSAGE, "|-:utf8", $sendmail ) { print MESSAGE $header, "\n", $body; close MESSAGE; }
I am currently playing with a line that has the following comment: my $val = encode( 'MIME-Q', $value ); # a nasty hack to fix Encode's wrapping 'feature':
Could be promising. It is applied to message headers.
Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://mailfilter.sunet.se/canit/urlproxy.php?_q=aHR0cHM6Ly9pZGVhcy5yZX BlYy5vcmcvemltbS8%3D&_r=b3J1LXNl&_s=c2tuQG9ydS5zZQ%3D%3D @CZimm_economist
On Sun, 10 Apr 2016, Dan Hayes wrote:
the mail gateway "mx1.stlouisfed.org" is configured in sendmail and postfix config files, /etc/mail/sendmail.mc and /etc/postfix/main.cf. I'm not sure where it might be included in perl code or why it would be.
On Sun, Apr 10, 2016 at 2:47 PM, Sune Karlsson <Sune.Karlsson@oru.se> wrote:
Some thoughts off the top of my head. Most probably random noise or things you have already thought of.
Is mx1.stlouisfed.org your mail gateway? Is this configured somewhere or picked up automatically? If configured it might help to trace where this variable is used.
Perl module or system call for sending e-mail? I doubt that something is "messing" with the e-mail address. It is more likely that whatever is sending the e-mail tacks on the @mx1.stlouisfed.org to the address for some strange reason.
/Sune
-----Original Message----- From: 'Christian Zimmermann' [mailto:zimmermann@stlouisfed.org] Sent: Sunday, April 10, 2016 8:03 PM To: Sune Karlsson Cc: Thomas Krichel; RAS-run Mailing List; Hayes, Dan -- Dan Hayes Subject: RE: [RAS] ReDIF-perl trouble
The latest of the little I was able to figure out.
ReDIF-perl does not seem to be to blame, at least not directly. Updating it triggered all sort of other updates, and despite reverting ReDIF- perl, emails are *not* going out. New registrants and email changes bounced hopping within our system.
Emails are typically formatted like zimmermann@stlouisfed.org@mx1.stlouisfed.org
but I have seen one split in two like this: christian.zimmermann@stls.frb.o@mx1.stlouisfed.org rg@mx1.stlouisfed.org
which makes absolutely no sense.
There must be some package that has started messing with those email addresses. From the list of updated packages, there is no obvious candidate.
Still digging.
And, no, Thomas I cannot give you access to the machine.
Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA
https://mailfilter.sunet.se/canit/urlproxy.php?_q=aHR0cHM6Ly9pZGVhcy5yZX
BlYy5vcmcvemltbS8%3D&_r=b3J1LXNl&_s=c2tuQG9ydS5zZQ%3D%3D @CZimm_economist
On Sun, 10 Apr 2016, Sune Karlsson wrote:
>> 2) I notice today the ReDIF-perl was not installed on RAS. after >> installation though, emails have broken. the to field is malformed, > for >> example: >> >> The following message to <"Christian Zimmermann" >> <zimmermann@stlouisfed.org>@mx1.stlouisfed.org> was >> undeliverable. >> >> returning to 2.70 fixed the problem, so it has nothing to do with any >> external perl module that refreshed during install. I suspect this has > to do >> with the following in Parser.pm: >> >> < 'remove_newline_from_values' => 0, >>> #SK 'remove_newline_from_values' => 0, Obsoleted by new >> Unicode processing, newlines are always removed >> >> or at least that is all I can see.
There are many differences between 2.70 and 2.74. It's not clear to me that they should cause problems with sending e-mails. At least I have not experienced any problems.
Which process is sending the e-mail, where does it get the e-mail address? Perl module for sending e-mail or system call to an external process? I suppose an assumption of a newline (or white space) at the end of the e-mail address that no longer holds could cause problems when building a command line.
/Sune
And it turns out all my playing with the .pm is for naught, as they stay in memory and are not read. What a nice Sunday I spent. Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist On Sun, 10 Apr 2016, 'Christian Zimmermann' wrote:
There is
print MESSAGE $header, "\n", $body;
I have not yet been successful in printing to file. I have to wait for each email to reach max hops to see a result... Not the most elegant and quite frustrating.
I am attaching what I think is the critical file, if that helps.
Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist
On Sun, 10 Apr 2016, Sune Karlsson wrote:
One thing that could be causing problems is if $header doesn't end with a newline. Then there would be no blank line between the headers and the body of the message.
I doubt that encode would do something really strange. Have you tried printing to a file instead of piping?
/Sune
-----Original Message----- From: 'Christian Zimmermann' [mailto:zimmermann@stlouisfed.org] Sent: Sunday, April 10, 2016 11:52 PM To: Dan Hayes Cc: Sune Karlsson; Thomas Krichel; RAS-run Mailing List Subject: Re: [RAS] ReDIF-perl trouble
The configuration file for ACIS (the software for RAS) has:
sendmail = '/usr/sbin/sendmail -t -f authors@repec.org'
So it is not using a perl module to send email. It pipes to sendmail:
my $sendmail = $config -> {sendmail}; if ( open MESSAGE, "|-:utf8", $sendmail ) { print MESSAGE $header, "\n", $body; close MESSAGE; }
I am currently playing with a line that has the following comment: my $val = encode( 'MIME-Q', $value ); # a nasty hack to fix Encode's wrapping 'feature':
Could be promising. It is applied to message headers.
Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://mailfilter.sunet.se/canit/urlproxy.php?_q=aHR0cHM6Ly9pZGVhcy5yZX BlYy5vcmcvemltbS8%3D&_r=b3J1LXNl&_s=c2tuQG9ydS5zZQ%3D%3D @CZimm_economist
On Sun, 10 Apr 2016, Dan Hayes wrote:
the mail gateway "mx1.stlouisfed.org" is configured in sendmail and postfix config files, /etc/mail/sendmail.mc and /etc/postfix/main.cf. I'm not sure where it might be included in perl code or why it would be.
On Sun, Apr 10, 2016 at 2:47 PM, Sune Karlsson <Sune.Karlsson@oru.se> wrote:
Some thoughts off the top of my head. Most probably random noise or things you have already thought of.
Is mx1.stlouisfed.org your mail gateway? Is this configured somewhere or picked up automatically? If configured it might help to trace where this variable is used.
Perl module or system call for sending e-mail? I doubt that something is "messing" with the e-mail address. It is more likely that whatever is sending the e-mail tacks on the @mx1.stlouisfed.org to the address for some strange reason.
/Sune
-----Original Message----- From: 'Christian Zimmermann' [mailto:zimmermann@stlouisfed.org] Sent: Sunday, April 10, 2016 8:03 PM To: Sune Karlsson Cc: Thomas Krichel; RAS-run Mailing List; Hayes, Dan -- Dan Hayes Subject: RE: [RAS] ReDIF-perl trouble
The latest of the little I was able to figure out.
ReDIF-perl does not seem to be to blame, at least not directly. Updating it triggered all sort of other updates, and despite reverting ReDIF- perl, emails are *not* going out. New registrants and email changes bounced hopping within our system.
Emails are typically formatted like zimmermann@stlouisfed.org@mx1.stlouisfed.org
but I have seen one split in two like this: christian.zimmermann@stls.frb.o@mx1.stlouisfed.org rg@mx1.stlouisfed.org
which makes absolutely no sense.
There must be some package that has started messing with those email addresses. From the list of updated packages, there is no obvious candidate.
Still digging.
And, no, Thomas I cannot give you access to the machine.
Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA
https://mailfilter.sunet.se/canit/urlproxy.php?_q=aHR0cHM6Ly9pZGVhcy5yZX
BlYy5vcmcvemltbS8%3D&_r=b3J1LXNl&_s=c2tuQG9ydS5zZQ%3D%3D @CZimm_economist
On Sun, 10 Apr 2016, Sune Karlsson wrote:
> > > 2) I notice today the ReDIF-perl was not installed on RAS. > > > after > > > installation though, emails have broken. the to field is malformed, > > for > > > example: > > > > > > The following message to <"Christian Zimmermann" > > > <zimmermann@stlouisfed.org>@mx1.stlouisfed.org> was > > > undeliverable. > > > > > > returning to 2.70 fixed the problem, so it has nothing to do > > > with any > > > external perl module that refreshed during install. I > > > suspect this has > > to do > > > with the following in Parser.pm: > > > > > > < 'remove_newline_from_values' => 0, > > > > #SK 'remove_newline_from_values' => 0, > > > > Obsoleted by new > > > Unicode processing, newlines are always removed > > > > > > or at least that is all I can see. > > There are many differences between 2.70 and 2.74. It's not clear > to me that they should cause problems with sending e-mails. At least I have not experienced any problems. > > Which process is sending the e-mail, where does it get the > e-mail address? Perl module for sending e-mail or system call to an external process? I suppose an assumption of a newline (or white space) at the end of the e-mail address that no longer holds could cause problems when building a command line. > > /Sune >
That is likely because starman is running the site, and I cannot find any documentation on how I can get it to source modified .pm files. On Sun, 10 Apr 2016, 'Christian Zimmermann' wrote:
And it turns out all my playing with the .pm is for naught, as they stay in memory and are not read. What a nice Sunday I spent.
Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist
On Sun, 10 Apr 2016, 'Christian Zimmermann' wrote:
There is
print MESSAGE $header, "\n", $body;
I have not yet been successful in printing to file. I have to wait for each email to reach max hops to see a result... Not the most elegant and quite frustrating.
I am attaching what I think is the critical file, if that helps.
Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist
On Sun, 10 Apr 2016, Sune Karlsson wrote:
One thing that could be causing problems is if $header doesn't end with a newline. Then there would be no blank line between the headers and the body of the message.
I doubt that encode would do something really strange. Have you tried printing to a file instead of piping?
/Sune
-----Original Message----- From: 'Christian Zimmermann' [mailto:zimmermann@stlouisfed.org] Sent: Sunday, April 10, 2016 11:52 PM To: Dan Hayes Cc: Sune Karlsson; Thomas Krichel; RAS-run Mailing List Subject: Re: [RAS] ReDIF-perl trouble
The configuration file for ACIS (the software for RAS) has:
sendmail = '/usr/sbin/sendmail -t -f authors@repec.org'
So it is not using a perl module to send email. It pipes to sendmail:
my $sendmail = $config -> {sendmail}; if ( open MESSAGE, "|-:utf8", $sendmail ) { print MESSAGE $header, "\n", $body; close MESSAGE; }
I am currently playing with a line that has the following comment: my $val = encode( 'MIME-Q', $value ); # a nasty hack to fix Encode's wrapping 'feature':
Could be promising. It is applied to message headers.
Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://mailfilter.sunet.se/canit/urlproxy.php?_q=aHR0cHM6Ly9pZGVhcy5yZX BlYy5vcmcvemltbS8%3D&_r=b3J1LXNl&_s=c2tuQG9ydS5zZQ%3D%3D @CZimm_economist
On Sun, 10 Apr 2016, Dan Hayes wrote:
the mail gateway "mx1.stlouisfed.org" is configured in sendmail and postfix config files, /etc/mail/sendmail.mc and /etc/postfix/main.cf. I'm not sure where it might be included in perl code or why it would be.
On Sun, Apr 10, 2016 at 2:47 PM, Sune Karlsson <Sune.Karlsson@oru.se> wrote:
Some thoughts off the top of my head. Most probably random noise or things you have already thought of.
Is mx1.stlouisfed.org your mail gateway? Is this configured somewhere or picked up automatically? If configured it might help to trace where this variable is used.
Perl module or system call for sending e-mail? I doubt that something is "messing" with the e-mail address. It is more likely that whatever is sending the e-mail tacks on the @mx1.stlouisfed.org to the address for some strange reason.
/Sune
> -----Original Message----- > From: 'Christian Zimmermann' [mailto:zimmermann@stlouisfed.org] > Sent: Sunday, April 10, 2016 8:03 PM > To: Sune Karlsson > Cc: Thomas Krichel; RAS-run Mailing List; Hayes, Dan -- Dan > Hayes > Subject: RE: [RAS] ReDIF-perl trouble > > The latest of the little I was able to figure out. > > ReDIF-perl does not seem to be to blame, at least not directly. Updating > it triggered all sort of other updates, and despite reverting > ReDIF- > perl, > emails are *not* going out. New registrants and email changes bounced > hopping within our system. > > Emails are typically formatted like > zimmermann@stlouisfed.org@mx1.stlouisfed.org > > but I have seen one split in two like this: > christian.zimmermann@stls.frb.o@mx1.stlouisfed.org > rg@mx1.stlouisfed.org > > which makes absolutely no sense. > > There must be some package that has started messing with those > email > addresses. From the list of updated packages, there is no > obvious > candidate. > > Still digging. > > And, no, Thomas I cannot give you access to the machine. > > > > Christian Zimmermann FIGUGEGL! > Economic Research > Federal Reserve Bank of St. Louis > P.O. Box 442 > St. Louis MO 63166-0442 USA > https://mailfilter.sunet.se/canit/urlproxy.php?_q=aHR0cHM6Ly9pZGVhcy5yZX > BlYy5vcmcvemltbS8%3D&_r=b3J1LXNl&_s=c2tuQG9ydS5zZQ%3D%3D > @CZimm_economist > > On Sun, 10 Apr 2016, Sune Karlsson wrote: > > > > > 2) I notice today the ReDIF-perl was not installed on > > > > RAS. after > > > > installation though, emails have broken. the to field is malformed, > > > for > > > > example: > > > > > > > > The following message to <"Christian Zimmermann" > > > > <zimmermann@stlouisfed.org>@mx1.stlouisfed.org> was > > > > undeliverable. > > > > > > > > returning to 2.70 fixed the problem, so it has nothing to > > > > do with > any > > > > external perl module that refreshed during install. I > > > > suspect this > has > > > to do > > > > with the following in Parser.pm: > > > > > > > > < 'remove_newline_from_values' => 0, > > > > > #SK 'remove_newline_from_values' => 0, > > > > > Obsoleted by new > > > > Unicode processing, newlines are always removed > > > > > > > > or at least that is all I can see. > > > > There are many differences between 2.70 and 2.74. It's not > > clear to me > that they should cause problems with sending e-mails. At least > I have > not experienced any problems. > > > > Which process is sending the e-mail, where does it get the > > e-mail > address? Perl module for sending e-mail or system call to an external > process? I suppose an assumption of a newline (or white space) > at the > end of the e-mail address that no longer holds could cause > problems when > building a command line. > > > > /Sune > >
From http://search.cpan.org/~miyagawa/Starman/script/starman
===================================================================== HUP Sending HUP signal to the master process will restart all the workers gracefully (meaning the currently running requests will shut down once the request is complete), and by default, the workers will pick up the code changes you make by reloading the application. If you enable --preload-app option, however, the code will be only loaded in the startup process and will not pick up the code changes you made. And --preload-app This option lets Starman preload the specified PSGI application in the master parent process before preforking children. This allows memory savings with copy-on-write memory management. When not set (default), forked children loads the application in the initialization hook. Enabling this option can cause bad things happen when resources like sockets or database connections are opened at load time by the master process and shared by multiple children. Since Starman 0.2000, this option defaults to false, and you should explicitly set this option to preload the application in the master process. Alternatively, you can use -M command line option (plackup's common option) to preload the modules rather than the <application> itself. starman -MCatalyst -MDBIx::Class myapp.psgi will load the modules in the master process for memory savings with CoW, but the actual loading of myapp.psgi is done per children, allowing resource managements such as database connection safer. If you enable this option, sending HUP signal to the master process will not pick up any code changes you make. See "SIGNALS" for details. ========================================================= If I am reading this correctly you need to restart Starman without the --preload-app or -M options if they are used. You can then use HUP. To see the effect of code changes you should probably run it with as few workers as possible. On the Email.pm module: - I think the nasty hack comment refers to the line after the comment. - I would try the built in debug/logging features. Enabling debug-email-data-log in the config, setting $debug to true in the code and instead of replacing the pipe to sendmail with opening a file adding something like if ( $debug_email_data_log ) { if ( open F, ">>:utf8", $debug_email_data_log ) { print F scalar( localtime ), "\n"; print F $header, "\n", $body; close F; } } This way you can see both what happens to the e-mail and (hopefully) what sendmail gets. Just my two cents. /Sune
-----Original Message----- From: 'Christian Zimmermann' [mailto:zimmermann@stlouisfed.org] Sent: Monday, April 11, 2016 2:19 AM To: Sune Karlsson Cc: RAS-run Mailing List Subject: Re: [RAS] ReDIF-perl trouble
That is likely because starman is running the site, and I cannot find any documentation on how I can get it to source modified .pm files.
On Sun, 10 Apr 2016, 'Christian Zimmermann' wrote:
And it turns out all my playing with the .pm is for naught, as they stay in memory and are not read. What a nice Sunday I spent.
Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA
https://mailfilter.sunet.se/canit/urlproxy.php?_q=aHR0cHM6Ly9pZGVhcy5yZX BlYy5vcmcvemltbS8%3D&_r=b3J1LXNl&_s=c2tuQG9ydS5zZQ%3D%3D @CZimm_economist
On Sun, 10 Apr 2016, 'Christian Zimmermann' wrote:
There is
print MESSAGE $header, "\n", $body;
I have not yet been successful in printing to file. I have to wait
for
each email to reach max hops to see a result... Not the most elegant and quite frustrating.
I am attaching what I think is the critical file, if that helps.
Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA
https://mailfilter.sunet.se/canit/urlproxy.php?_q=aHR0cHM6Ly9pZGVhcy5yZX BlYy5vcmcvemltbS8%3D&_r=b3J1LXNl&_s=c2tuQG9ydS5zZQ%3D%3D @CZimm_economist
On Sun, 10 Apr 2016, Sune Karlsson wrote:
One thing that could be causing problems is if $header doesn't
end with
a newline. Then there would be no blank line between the headers and the body of the message.
I doubt that encode would do something really strange. Have you tried printing to a file instead of piping?
/Sune
-----Original Message----- From: 'Christian Zimmermann' [mailto:zimmermann@stlouisfed.org] Sent: Sunday, April 10, 2016 11:52 PM To: Dan Hayes Cc: Sune Karlsson; Thomas Krichel; RAS-run Mailing List Subject: Re: [RAS] ReDIF-perl trouble
The configuration file for ACIS (the software for RAS) has:
sendmail = '/usr/sbin/sendmail -t -f authors@repec.org'
So it is not using a perl module to send email. It pipes to sendmail:
my $sendmail = $config -> {sendmail}; if ( open MESSAGE, "|-:utf8", $sendmail ) { print MESSAGE $header, "\n", $body; close MESSAGE; }
I am currently playing with a line that has the following comment: my $val = encode( 'MIME-Q', $value ); # a nasty hack to fix Encode's wrapping 'feature':
Could be promising. It is applied to message headers.
Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA
https://mailfilter.sunet.se/canit/urlproxy.php?_q=aHR0cHM6Ly9pZGVhcy5yZX
BlYy5vcmcvemltbS8%3D&_r=b3J1LXNl&_s=c2tuQG9ydS5zZQ%3D%3D @CZimm_economist
On Sun, 10 Apr 2016, Dan Hayes wrote:
the mail gateway "mx1.stlouisfed.org" is configured in sendmail and postfix config files, /etc/mail/sendmail.mc and /etc/postfix/main.cf. I'm not sure where it might be included in perl code or why it would be.
On Sun, Apr 10, 2016 at 2:47 PM, Sune Karlsson <Sune.Karlsson@oru.se> wrote:
> Some thoughts off the top of my head. Most probably random noise > or things > you have already thought of. > > Is mx1.stlouisfed.org your mail gateway? Is this configured > somewhere or > picked up automatically? If configured it might help to trace > where this > variable is used. > > Perl module or system call for sending e-mail? I doubt that > something is > "messing" with the e-mail address. It is more likely that > whatever is > sending the e-mail tacks on the @mx1.stlouisfed.org to the > address for > some strange reason. > > /Sune > > > -----Original Message----- > > From: 'Christian Zimmermann' [mailto:zimmermann@stlouisfed.org] > > Sent: Sunday, April 10, 2016 8:03 PM > > To: Sune Karlsson > > Cc: Thomas Krichel; RAS-run Mailing List; Hayes, Dan -- Dan > > Hayes > > Subject: RE: [RAS] ReDIF-perl trouble > > > > The latest of the little I was able to figure out. > > > > ReDIF-perl does not seem to be to blame, at least not directly. Updating > > it triggered all sort of other updates, and despite reverting > > ReDIF- > > perl, > > emails are *not* going out. New registrants and email changes bounced > > hopping within our system. > > > > Emails are typically formatted like > > zimmermann@stlouisfed.org@mx1.stlouisfed.org > > > > but I have seen one split in two like this: > > christian.zimmermann@stls.frb.o@mx1.stlouisfed.org > > rg@mx1.stlouisfed.org > > > > which makes absolutely no sense. > > > > There must be some package that has started messing with those > > email > > addresses. From the list of updated packages, there is no > > obvious > > candidate. > > > > Still digging. > > > > And, no, Thomas I cannot give you access to the machine. > > > > > > > > Christian Zimmermann FIGUGEGL! > > Economic Research > > Federal Reserve Bank of St. Louis > > P.O. Box 442 > > St. Louis MO 63166-0442 USA > >
https://mailfilter.sunet.se/canit/urlproxy.php?_q=aHR0cHM6Ly9pZGVhcy5yZX
> > BlYy5vcmcvemltbS8%3D&_r=b3J1LXNl&_s=c2tuQG9ydS5zZQ%3D%3D > > @CZimm_economist > > > > On Sun, 10 Apr 2016, Sune Karlsson wrote: > > > > > > > 2) I notice today the ReDIF-perl was not installed on > > > > > RAS. after > > > > > installation though, emails have broken. the to field is malformed, > > > > for > > > > > example: > > > > > > > > > > The following message to <"Christian Zimmermann" > > > > > <zimmermann@stlouisfed.org>@mx1.stlouisfed.org> was > > > > > undeliverable. > > > > > > > > > > returning to 2.70 fixed the problem, so it has nothing to > > > > > do with > > any > > > > > external perl module that refreshed during install. I > > > > > suspect this > > has > > > > to do > > > > > with the following in Parser.pm: > > > > > > > > > > < 'remove_newline_from_values' => 0, > > > > > > #SK 'remove_newline_from_values' => 0, > > > > > > Obsoleted by new > > > > > Unicode processing, newlines are always removed > > > > > > > > > > or at least that is all I can see. > > > > > > There are many differences between 2.70 and 2.74. It's not > > > clear to me > > that they should cause problems with sending e-mails. At least > > I have > > not experienced any problems. > > > > > > Which process is sending the e-mail, where does it get the > > > e-mail > > address? Perl module for sending e-mail or system call to an external > > process? I suppose an assumption of a newline (or white space) > > at the > > end of the e-mail address that no longer holds could cause > > problems when > > building a command line. > > > > > > /Sune > > > >
You are likely correct on all counts, but I want to restart starman with a sysadmin next to me in case things go wrong. ACIS is a beast that takes forever to restart. Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist On Mon, 11 Apr 2016, Sune Karlsson wrote:
From http://search.cpan.org/~miyagawa/Starman/script/starman
=====================================================================
HUP
Sending HUP signal to the master process will restart all the workers gracefully (meaning the currently running requests will shut down once the request is complete), and by default, the workers will pick up the code changes you make by reloading the application.
If you enable --preload-app option, however, the code will be only loaded in the startup process and will not pick up the code changes you made.
And
--preload-app
This option lets Starman preload the specified PSGI application in the master parent process before preforking children. This allows memory savings with copy-on-write memory management. When not set (default), forked children loads the application in the initialization hook.
Enabling this option can cause bad things happen when resources like sockets or database connections are opened at load time by the master process and shared by multiple children.
Since Starman 0.2000, this option defaults to false, and you should explicitly set this option to preload the application in the master process.
Alternatively, you can use -M command line option (plackup's common option) to preload the modules rather than the <application> itself.
starman -MCatalyst -MDBIx::Class myapp.psgi
will load the modules in the master process for memory savings with CoW, but the actual loading of myapp.psgi is done per children, allowing resource managements such as database connection safer.
If you enable this option, sending HUP signal to the master process will not pick up any code changes you make. See "SIGNALS" for details.
=========================================================
If I am reading this correctly you need to restart Starman without the --preload-app or -M options if they are used. You can then use HUP. To see the effect of code changes you should probably run it with as few workers as possible.
On the Email.pm module:
- I think the nasty hack comment refers to the line after the comment. - I would try the built in debug/logging features. Enabling debug-email-data-log in the config, setting $debug to true in the code and instead of replacing the pipe to sendmail with opening a file adding something like if ( $debug_email_data_log ) { if ( open F, ">>:utf8", $debug_email_data_log ) { print F scalar( localtime ), "\n"; print F $header, "\n", $body; close F; } } This way you can see both what happens to the e-mail and (hopefully) what sendmail gets.
Just my two cents.
/Sune
-----Original Message----- From: 'Christian Zimmermann' [mailto:zimmermann@stlouisfed.org] Sent: Monday, April 11, 2016 2:19 AM To: Sune Karlsson Cc: RAS-run Mailing List Subject: Re: [RAS] ReDIF-perl trouble
That is likely because starman is running the site, and I cannot find any documentation on how I can get it to source modified .pm files.
On Sun, 10 Apr 2016, 'Christian Zimmermann' wrote:
And it turns out all my playing with the .pm is for naught, as they stay in memory and are not read. What a nice Sunday I spent.
Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA
https://mailfilter.sunet.se/canit/urlproxy.php?_q=aHR0cHM6Ly9pZGVhcy5yZX BlYy5vcmcvemltbS8%3D&_r=b3J1LXNl&_s=c2tuQG9ydS5zZQ%3D%3D @CZimm_economist
On Sun, 10 Apr 2016, 'Christian Zimmermann' wrote:
There is
print MESSAGE $header, "\n", $body;
I have not yet been successful in printing to file. I have to wait
for
each email to reach max hops to see a result... Not the most elegant and quite frustrating.
I am attaching what I think is the critical file, if that helps.
Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA
https://mailfilter.sunet.se/canit/urlproxy.php?_q=aHR0cHM6Ly9pZGVhcy5yZX BlYy5vcmcvemltbS8%3D&_r=b3J1LXNl&_s=c2tuQG9ydS5zZQ%3D%3D @CZimm_economist
On Sun, 10 Apr 2016, Sune Karlsson wrote:
One thing that could be causing problems is if $header doesn't
end with
a newline. Then there would be no blank line between the headers and the body of the message.
I doubt that encode would do something really strange. Have you tried printing to a file instead of piping?
/Sune
-----Original Message----- From: 'Christian Zimmermann' [mailto:zimmermann@stlouisfed.org] Sent: Sunday, April 10, 2016 11:52 PM To: Dan Hayes Cc: Sune Karlsson; Thomas Krichel; RAS-run Mailing List Subject: Re: [RAS] ReDIF-perl trouble
The configuration file for ACIS (the software for RAS) has:
sendmail = '/usr/sbin/sendmail -t -f authors@repec.org'
So it is not using a perl module to send email. It pipes to sendmail:
my $sendmail = $config -> {sendmail}; if ( open MESSAGE, "|-:utf8", $sendmail ) { print MESSAGE $header, "\n", $body; close MESSAGE; }
I am currently playing with a line that has the following comment: my $val = encode( 'MIME-Q', $value ); # a nasty hack to fix Encode's wrapping 'feature':
Could be promising. It is applied to message headers.
Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA
https://mailfilter.sunet.se/canit/urlproxy.php?_q=aHR0cHM6Ly9pZGVhcy5yZX
BlYy5vcmcvemltbS8%3D&_r=b3J1LXNl&_s=c2tuQG9ydS5zZQ%3D%3D @CZimm_economist
On Sun, 10 Apr 2016, Dan Hayes wrote:
> the mail gateway "mx1.stlouisfed.org" is configured in sendmail and postfix > config files, /etc/mail/sendmail.mc and /etc/postfix/main.cf. I'm > not sure > where it might be included in perl code or why it would be. > > On Sun, Apr 10, 2016 at 2:47 PM, Sune Karlsson > <Sune.Karlsson@oru.se> wrote: > >> Some thoughts off the top of my head. Most probably random noise >> or things >> you have already thought of. >> >> Is mx1.stlouisfed.org your mail gateway? Is this configured >> somewhere or >> picked up automatically? If configured it might help to trace >> where this >> variable is used. >> >> Perl module or system call for sending e-mail? I doubt that >> something is >> "messing" with the e-mail address. It is more likely that >> whatever is >> sending the e-mail tacks on the @mx1.stlouisfed.org to the >> address for >> some strange reason. >> >> /Sune >> >>> -----Original Message----- >>> From: 'Christian Zimmermann' [mailto:zimmermann@stlouisfed.org] >>> Sent: Sunday, April 10, 2016 8:03 PM >>> To: Sune Karlsson >>> Cc: Thomas Krichel; RAS-run Mailing List; Hayes, Dan -- Dan >>> Hayes >>> Subject: RE: [RAS] ReDIF-perl trouble >>> >>> The latest of the little I was able to figure out. >>> >>> ReDIF-perl does not seem to be to blame, at least not directly. Updating >>> it triggered all sort of other updates, and despite reverting >>> ReDIF- >>> perl, >>> emails are *not* going out. New registrants and email changes bounced >>> hopping within our system. >>> >>> Emails are typically formatted like >>> zimmermann@stlouisfed.org@mx1.stlouisfed.org >>> >>> but I have seen one split in two like this: >>> christian.zimmermann@stls.frb.o@mx1.stlouisfed.org >>> rg@mx1.stlouisfed.org >>> >>> which makes absolutely no sense. >>> >>> There must be some package that has started messing with those >>> email >>> addresses. From the list of updated packages, there is no >>> obvious >>> candidate. >>> >>> Still digging. >>> >>> And, no, Thomas I cannot give you access to the machine. >>> >>> >>> >>> Christian Zimmermann FIGUGEGL! >>> Economic Research >>> Federal Reserve Bank of St. Louis >>> P.O. Box 442 >>> St. Louis MO 63166-0442 USA >>>
https://mailfilter.sunet.se/canit/urlproxy.php?_q=aHR0cHM6Ly9pZGVhcy5yZX
>>> BlYy5vcmcvemltbS8%3D&_r=b3J1LXNl&_s=c2tuQG9ydS5zZQ%3D%3D >>> @CZimm_economist >>> >>> On Sun, 10 Apr 2016, Sune Karlsson wrote: >>> >>>>>> 2) I notice today the ReDIF-perl was not installed on >>>>>> RAS. after >>>>>> installation though, emails have broken. the to field is malformed, >>>>> for >>>>>> example: >>>>>> >>>>>> The following message to <"Christian Zimmermann" >>>>>> <zimmermann@stlouisfed.org>@mx1.stlouisfed.org> was >>>>>> undeliverable. >>>>>> >>>>>> returning to 2.70 fixed the problem, so it has nothing to >>>>>> do with >>> any >>>>>> external perl module that refreshed during install. I >>>>>> suspect this >>> has >>>>> to do >>>>>> with the following in Parser.pm: >>>>>> >>>>>> < 'remove_newline_from_values' => 0, >>>>>>> #SK 'remove_newline_from_values' => 0, >>>>>>> Obsoleted by new >>>>>> Unicode processing, newlines are always removed >>>>>> >>>>>> or at least that is all I can see. >>>> >>>> There are many differences between 2.70 and 2.74. It's not >>>> clear to me >>> that they should cause problems with sending e-mails. At least >>> I have >>> not experienced any problems. >>>> >>>> Which process is sending the e-mail, where does it get the >>>> e-mail >>> address? Perl module for sending e-mail or system call to an external >>> process? I suppose an assumption of a newline (or white space) >>> at the >>> end of the e-mail address that no longer holds could cause >>> problems when >>> building a command line. >>>> >>>> /Sune >>>> >> >
Update. It is working now, and has for about 14 hours. We found that removing all encoding from the headers was the solution. It seems to work even for authors with extended-ASCII names. Starman was a big problem though. We ended up having to reboot for every configuration we wanted to try. Nothing else would work. And some reboots would not work. We are cognisant now that we will live in Linux hell for all these reboots. Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA https://ideas.repec.org/zimm/ @CZimm_economist On Mon, 11 Apr 2016, Sune Karlsson wrote:
From http://search.cpan.org/~miyagawa/Starman/script/starman
=====================================================================
HUP
Sending HUP signal to the master process will restart all the workers gracefully (meaning the currently running requests will shut down once the request is complete), and by default, the workers will pick up the code changes you make by reloading the application.
If you enable --preload-app option, however, the code will be only loaded in the startup process and will not pick up the code changes you made.
And
--preload-app
This option lets Starman preload the specified PSGI application in the master parent process before preforking children. This allows memory savings with copy-on-write memory management. When not set (default), forked children loads the application in the initialization hook.
Enabling this option can cause bad things happen when resources like sockets or database connections are opened at load time by the master process and shared by multiple children.
Since Starman 0.2000, this option defaults to false, and you should explicitly set this option to preload the application in the master process.
Alternatively, you can use -M command line option (plackup's common option) to preload the modules rather than the <application> itself.
starman -MCatalyst -MDBIx::Class myapp.psgi
will load the modules in the master process for memory savings with CoW, but the actual loading of myapp.psgi is done per children, allowing resource managements such as database connection safer.
If you enable this option, sending HUP signal to the master process will not pick up any code changes you make. See "SIGNALS" for details.
=========================================================
If I am reading this correctly you need to restart Starman without the --preload-app or -M options if they are used. You can then use HUP. To see the effect of code changes you should probably run it with as few workers as possible.
On the Email.pm module:
- I think the nasty hack comment refers to the line after the comment. - I would try the built in debug/logging features. Enabling debug-email-data-log in the config, setting $debug to true in the code and instead of replacing the pipe to sendmail with opening a file adding something like if ( $debug_email_data_log ) { if ( open F, ">>:utf8", $debug_email_data_log ) { print F scalar( localtime ), "\n"; print F $header, "\n", $body; close F; } } This way you can see both what happens to the e-mail and (hopefully) what sendmail gets.
Just my two cents.
/Sune
-----Original Message----- From: 'Christian Zimmermann' [mailto:zimmermann@stlouisfed.org] Sent: Monday, April 11, 2016 2:19 AM To: Sune Karlsson Cc: RAS-run Mailing List Subject: Re: [RAS] ReDIF-perl trouble
That is likely because starman is running the site, and I cannot find any documentation on how I can get it to source modified .pm files.
On Sun, 10 Apr 2016, 'Christian Zimmermann' wrote:
And it turns out all my playing with the .pm is for naught, as they stay in memory and are not read. What a nice Sunday I spent.
Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA
https://mailfilter.sunet.se/canit/urlproxy.php?_q=aHR0cHM6Ly9pZGVhcy5yZX BlYy5vcmcvemltbS8%3D&_r=b3J1LXNl&_s=c2tuQG9ydS5zZQ%3D%3D @CZimm_economist
On Sun, 10 Apr 2016, 'Christian Zimmermann' wrote:
There is
print MESSAGE $header, "\n", $body;
I have not yet been successful in printing to file. I have to wait
for
each email to reach max hops to see a result... Not the most elegant and quite frustrating.
I am attaching what I think is the critical file, if that helps.
Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA
https://mailfilter.sunet.se/canit/urlproxy.php?_q=aHR0cHM6Ly9pZGVhcy5yZX BlYy5vcmcvemltbS8%3D&_r=b3J1LXNl&_s=c2tuQG9ydS5zZQ%3D%3D @CZimm_economist
On Sun, 10 Apr 2016, Sune Karlsson wrote:
One thing that could be causing problems is if $header doesn't
end with
a newline. Then there would be no blank line between the headers and the body of the message.
I doubt that encode would do something really strange. Have you tried printing to a file instead of piping?
/Sune
-----Original Message----- From: 'Christian Zimmermann' [mailto:zimmermann@stlouisfed.org] Sent: Sunday, April 10, 2016 11:52 PM To: Dan Hayes Cc: Sune Karlsson; Thomas Krichel; RAS-run Mailing List Subject: Re: [RAS] ReDIF-perl trouble
The configuration file for ACIS (the software for RAS) has:
sendmail = '/usr/sbin/sendmail -t -f authors@repec.org'
So it is not using a perl module to send email. It pipes to sendmail:
my $sendmail = $config -> {sendmail}; if ( open MESSAGE, "|-:utf8", $sendmail ) { print MESSAGE $header, "\n", $body; close MESSAGE; }
I am currently playing with a line that has the following comment: my $val = encode( 'MIME-Q', $value ); # a nasty hack to fix Encode's wrapping 'feature':
Could be promising. It is applied to message headers.
Christian Zimmermann FIGUGEGL! Economic Research Federal Reserve Bank of St. Louis P.O. Box 442 St. Louis MO 63166-0442 USA
https://mailfilter.sunet.se/canit/urlproxy.php?_q=aHR0cHM6Ly9pZGVhcy5yZX
BlYy5vcmcvemltbS8%3D&_r=b3J1LXNl&_s=c2tuQG9ydS5zZQ%3D%3D @CZimm_economist
On Sun, 10 Apr 2016, Dan Hayes wrote:
> the mail gateway "mx1.stlouisfed.org" is configured in sendmail and postfix > config files, /etc/mail/sendmail.mc and /etc/postfix/main.cf. I'm > not sure > where it might be included in perl code or why it would be. > > On Sun, Apr 10, 2016 at 2:47 PM, Sune Karlsson > <Sune.Karlsson@oru.se> wrote: > >> Some thoughts off the top of my head. Most probably random noise >> or things >> you have already thought of. >> >> Is mx1.stlouisfed.org your mail gateway? Is this configured >> somewhere or >> picked up automatically? If configured it might help to trace >> where this >> variable is used. >> >> Perl module or system call for sending e-mail? I doubt that >> something is >> "messing" with the e-mail address. It is more likely that >> whatever is >> sending the e-mail tacks on the @mx1.stlouisfed.org to the >> address for >> some strange reason. >> >> /Sune >> >>> -----Original Message----- >>> From: 'Christian Zimmermann' [mailto:zimmermann@stlouisfed.org] >>> Sent: Sunday, April 10, 2016 8:03 PM >>> To: Sune Karlsson >>> Cc: Thomas Krichel; RAS-run Mailing List; Hayes, Dan -- Dan >>> Hayes >>> Subject: RE: [RAS] ReDIF-perl trouble >>> >>> The latest of the little I was able to figure out. >>> >>> ReDIF-perl does not seem to be to blame, at least not directly. Updating >>> it triggered all sort of other updates, and despite reverting >>> ReDIF- >>> perl, >>> emails are *not* going out. New registrants and email changes bounced >>> hopping within our system. >>> >>> Emails are typically formatted like >>> zimmermann@stlouisfed.org@mx1.stlouisfed.org >>> >>> but I have seen one split in two like this: >>> christian.zimmermann@stls.frb.o@mx1.stlouisfed.org >>> rg@mx1.stlouisfed.org >>> >>> which makes absolutely no sense. >>> >>> There must be some package that has started messing with those >>> email >>> addresses. From the list of updated packages, there is no >>> obvious >>> candidate. >>> >>> Still digging. >>> >>> And, no, Thomas I cannot give you access to the machine. >>> >>> >>> >>> Christian Zimmermann FIGUGEGL! >>> Economic Research >>> Federal Reserve Bank of St. Louis >>> P.O. Box 442 >>> St. Louis MO 63166-0442 USA >>>
https://mailfilter.sunet.se/canit/urlproxy.php?_q=aHR0cHM6Ly9pZGVhcy5yZX
>>> BlYy5vcmcvemltbS8%3D&_r=b3J1LXNl&_s=c2tuQG9ydS5zZQ%3D%3D >>> @CZimm_economist >>> >>> On Sun, 10 Apr 2016, Sune Karlsson wrote: >>> >>>>>> 2) I notice today the ReDIF-perl was not installed on >>>>>> RAS. after >>>>>> installation though, emails have broken. the to field is malformed, >>>>> for >>>>>> example: >>>>>> >>>>>> The following message to <"Christian Zimmermann" >>>>>> <zimmermann@stlouisfed.org>@mx1.stlouisfed.org> was >>>>>> undeliverable. >>>>>> >>>>>> returning to 2.70 fixed the problem, so it has nothing to >>>>>> do with >>> any >>>>>> external perl module that refreshed during install. I >>>>>> suspect this >>> has >>>>> to do >>>>>> with the following in Parser.pm: >>>>>> >>>>>> < 'remove_newline_from_values' => 0, >>>>>>> #SK 'remove_newline_from_values' => 0, >>>>>>> Obsoleted by new >>>>>> Unicode processing, newlines are always removed >>>>>> >>>>>> or at least that is all I can see. >>>> >>>> There are many differences between 2.70 and 2.74. It's not >>>> clear to me >>> that they should cause problems with sending e-mails. At least >>> I have >>> not experienced any problems. >>>> >>>> Which process is sending the e-mail, where does it get the >>>> e-mail >>> address? Perl module for sending e-mail or system call to an external >>> process? I suppose an assumption of a newline (or white space) >>> at the >>> end of the e-mail address that no longer holds could cause >>> problems when >>> building a command line. >>>> >>>> /Sune >>>> >> >
'Christian Zimmermann' writes
It is working now, and has for about 14 hours. We found that removing all encoding from the headers was the solution. It seems to work even for authors with extended-ASCII names.
Strange ....
Starman was a big problem though. We ended up having to reboot for every configuration we wanted to try. Nothing else would work. And some reboots would not work. We are cognisant now that we will live in Linux hell for all these reboots.
Well I did not come up with that idea, and I don't thnink starman is part of ACIS. For AuthorClaim, I did change the web serving to a plain fastGGI. It's what I do for ernad and icanis as well. It's rock solid. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
'Christian Zimmermann' writes
And, no, Thomas I cannot give you access to the machine.
Then you have the pleasure of fixing it on your own. I have done several months of work with ernad, on making improvements you did not use and now my expertise goes to waste as well. FWIW, I attach how Ernad sends mail. It's a bit more general as I produce both HTML and plain text attachments. I have three stylesheets, one for HTML, one for plain text and one for the header. The latter produces a simple XML, that I then parse. The wonderful thing is that I can debug the XSLT separately without having to touch the Perl. The code deals properly with non-Ascii header values. Thus we don't need to configure an all Ascii editor name any more. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
On Mon, 11 Apr 2016, Thomas Krichel wrote:
'Christian Zimmermann' writes
And, no, Thomas I cannot give you access to the machine.
Then you have the pleasure of fixing it on your own.
I have done several months of work with ernad, on making improvements you did not use and now my expertise goes to waste as well.
"you" as "me"? If so I do not understand.
FWIW, I attach how Ernad sends mail. It's a bit more general as I produce both HTML and plain text attachments. I have three stylesheets, one for HTML, one for plain text and one for the header. The latter produces a simple XML, that I then parse. The wonderful thing is that I can debug the XSLT separately without having to touch the Perl. The code deals properly with non-Ascii header values. Thus we don't need to configure an all Ascii editor name any more.
I see you use exim and encode as MIME-Header. RAS uses sendmail and encodes as MIME-Q. And indeed, I suspect the encoding is doing something funny. But before I can try successfully something, I need to figure out how I can restart starman gracefully after changing Email.pm.
--
Cheers,
Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
'Christian Zimmermann' writes
I have done several months of work with ernad, on making improvements you did not use and now my expertise goes to waste as well.
"you" as "me"?
Yes.
If so I do not understand.
Remember the machine learning I built into ernad? It is not in the version that runs on RAS.
I see you use exim and encode as MIME-Header. RAS uses sendmail
I think the -t is the same option between sendmail and exim so it should not make a difference. Ernad also saves a copy of the full email that is sent in a file.
I need to figure out how I can restart starman gracefully after changing Email.pm.
I have no idea what starman is. Google claims it's a David Bowie song. But I doubt you play music to servers so that they fix their bugs themselves. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
On Mon, 11 Apr 2016, Thomas Krichel wrote:
'Christian Zimmermann' writes
I have done several months of work with ernad, on making improvements you did not use and now my expertise goes to waste as well.
"you" as "me"?
Yes.
If so I do not understand.
Remember the machine learning I built into ernad? It is not in the version that runs on RAS.
I had no idea it was intended for RAS.
I see you use exim and encode as MIME-Header. RAS uses sendmail
I think the -t is the same option between sendmail and exim so it should not make a difference. Ernad also saves a copy of the full email that is sent in a file.
I would not want to do this of RAS. It is already extremely I/O intensive. We already have to isolate it on a separate physical server because of that.
I need to figure out how I can restart starman gracefully after changing Email.pm.
I have no idea what starman is. Google claims it's a David Bowie song. But I doubt you play music to servers so that they fix their bugs themselves.
http://search.cpan.org/~miyagawa/Starman/
--
Cheers,
Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
'Christian Zimmermann' writes
Remember the machine learning I built into ernad? It is not in the version that runs on RAS.
I had no idea it was intended for RAS.
Sorry, I meant to say "into acis". I still have to get my first coffee of the morning. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
'Christian Zimmermann' writes
I would not want to do this of RAS. It is already extremely I/O intensive.
Not all of acis is. Only the acis metadata updater is. You would get mileage by having that part be rewritten.
I don't recall this being used in ACIS. This must be your later addition. In summary, it appears very unlikely that ReDIF-perl is to blame here. -- Cheers, Thomas Krichel http://openlib.org/home/krichel skype:thomaskrichel
participants (4)
-
'Christian Zimmermann' -
Dan Hayes -
Sune Karlsson -
Thomas Krichel