15:00:11 <JBoyer> #startmeeting 2022-03-08 - Developer Meeting
15:00:11 <pinesol> Meeting started Tue Mar  8 15:00:11 2022 US/Eastern.  The chair is JBoyer. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:11 <pinesol> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:11 <pinesol> The meeting name has been set to '2022_03_08___developer_meeting'
15:00:15 <JBoyer> #info Agenda at https://wiki.evergreen-ils.org/doku.php?id=dev:meetings:2022-03-08
15:00:24 <JBoyer> #topic Introductions
15:00:25 <JBoyer> #info JBoyer  = Jason Boyer, EOLI
15:00:26 <abowling> #info abowling = Adam Bowling, Emerald Data Networks
15:00:33 <JBoyer> Whois <you lot>
15:00:46 <phasefx> #info phasefx = Jason Etheridge, EOLI
15:00:46 <dluch> #info dluch = Debbie Luchenbill, MOBIUS
15:00:48 <mmorgan> #info mmorgan = Michele Morgan, NOBLE
15:00:51 <csharp_> #info csharp = Chris Sharp, GPLS
15:00:51 <sandbergja> #info sandbergja = Jane Sandberg, Evergreen enthusiast
15:00:58 <collum> #info collum = Garry Collum, KCPL
15:01:02 <shulabear> #info shulabear = Shula Link, GCHRL in GAPines
15:01:09 <csharp_> sandbergja++
15:01:09 <berick> #info berick Bill Erickson, KCLS
15:01:16 <shulabear> sandbergja++
15:01:29 <csharp_> @who is an Evergreen enthusiast?
15:01:29 <pinesol> eglogbot is an Evergreen enthusiast.
15:01:29 <mmorgan> sandbergja++
15:01:34 <csharp_> pinesol: nah
15:01:34 <pinesol> csharp_: uh huh.. please tell me more about that
15:01:50 <dluch> Yay! Hi, sandbergja!!
15:02:06 <JBoyer> sandbergja++
15:02:09 <sandbergja> heya!
15:02:21 <JBoyer> Ok, people can feel free to #info-up as they arrive.
15:02:24 <terranm> #info terranm Terran McCanna, PINES
15:02:58 <JBoyer> Lots of placeholders today, so lets skip ahead a little.
15:03:07 <JBoyer> #topic Documentation
15:03:09 <JBoyer> #info Minutes from March 3 meeting can be found here: https://wiki.evergreen-ils.org/doku.php?id=evergreen-docs:dig_meetings
15:03:14 <JBoyer> #info We are updating and reorganizing the official docs and are going to need devs to check over some of the documentation for installation, configuration/customization, etc. to make sure it's complete and correct. So please keep an eye out for those asks!
15:03:27 <JBoyer> DIG folks, anything to add to those?
15:03:32 <dluch> Nope!
15:03:40 <JBoyer> dluch++
15:03:51 <sandbergja> dluch++
15:04:00 <sandbergja> dig++
15:04:02 <JBoyer> #topic Launchpad Status - snapshots
15:04:02 <JBoyer> #info Open Bugs - 2683
15:04:09 <JBoyer> #info Pullrequests - 115
15:04:09 <JBoyer> #info Signedoff - 44
15:04:18 <JBoyer> #topic Launchpad Status - Updates
15:04:18 <JBoyer> v
15:04:20 <JBoyer> #info Bugs Added - 35
15:04:29 <JBoyer> #info Pullrequest tag Added - 17
15:04:29 <JBoyer> #info Signedoff tag Added - 5
15:04:33 <JBoyer> #info Fix Committed - 10
15:04:47 <JBoyer> And finally on to
15:04:47 <JBoyer> #topic New Business
15:04:55 <JBoyer> #info LP 1787968 - How to track down errors for files that look like they should work
15:04:56 <pinesol> Launchpad bug 1787968 in Evergreen "Wishlist: Cover image uploader" [Wishlist,Confirmed] https://launchpad.net/bugs/1787968 - Assigned to Michele Morgan (mmorgan)
15:05:00 <JBoyer> mmorgan ?
15:05:24 <mmorgan> This is looking really good, but some uploads fail inexplicably
15:05:35 <mmorgan> Just looking for clues to figure out why.
15:05:57 <JBoyer> I ... think I opened the wrong bug then
15:06:01 <mmorgan> Can't find anything logged on the server.
15:06:27 <phasefx> mmorgan: do you have nginx in front of everything?
15:06:45 <mmorgan> phasefx: no, I don't
15:07:02 <csharp_> mmorgan: be sure to check the stderr logs in /openils/var/log too
15:07:07 <phasefx> k.  I've seen file limits enforced in nginx cause problems
15:07:39 <JBoyer> OH! I have had some fun with this on my personal instance. :/ In my case POSTs are being redirected with 301's, which then turns them into GETs which doesn't get it.
15:07:40 * mmorgan has checked the stderr logs and haven't seen anything there
15:07:58 <csharp_> oh yeah, could be similar to the request size thing we hit where ejabberd croks?
15:08:05 <csharp_> er... croaks
15:08:41 <JBoyer> mmorgan, do you have any proxy setup or do both apache and websocketd listen live on the external IP?
15:09:14 <mmorgan> JBoyer: no proxy involved
15:09:18 <JBoyer> Because I think apache can have an upload size limit but I don't know what the default is.
15:09:30 <phasefx> it's sad, but my debugging method of choice would be to sprinkle log output througout the code and see what spits out last
15:09:51 <mmorgan> Wouldn't it be logged somewhere if a limit were hit?
15:10:21 <JBoyer> Maybe. It's hard to say, and definitely wouldn't hit OpenSRF / Evergreen logs.
15:11:32 <shulabear> I want to say I've seen those pop in the apache logs when I've had them, but don't quote me on that.
15:11:39 <JBoyer> If you're using the Evergreen rsyslog config it could be in the apache logs, otherwise probably in /var/log/apache2/*
15:11:55 <csharp_> mmorgan: look into wireshark/tshark or tcpdump as the upload is happening
15:12:06 <phasefx> mmorgan: the uploads that cause problems are consistent? send me a link to a file and I can look later
15:12:15 <csharp_> (on the receiving server)
15:12:39 <phasefx> I can at least confirm or rule out whether it's the file or the environment
15:13:00 <mmorgan> phasefx: Yes, the failures are consistent.
15:13:25 * mmorgan will provide a link
15:13:30 <phasefx> mmorgan++
15:14:15 <mmorgan> So basically turn up logging to the max and scour every log, will give that a try.
15:14:58 <phasefx> transparent proxy (this file is a virus)
15:15:33 <csharp_> mmorgan: that was marginally useful for me - those logs are super dense
15:15:52 <csharp_> unless this is on a low-traffic server
15:15:54 * phasefx can't tell you how many times leaving the log details up has caused him to run out of disk space :)
15:16:00 <csharp_> oh yea
15:16:30 <mmorgan> test server, so should be ok :)
15:16:52 <mmorgan> Thanks for the tips!
15:16:56 <JBoyer> mmorgan++
15:16:59 <JBoyer> phasefx++
15:17:03 <JBoyer> csharp_++
15:17:15 <mmorgan> phasefx++
15:17:22 <JBoyer> Ok, any other new business or last minute bolts of inspiration lightning?
15:17:23 <mmorgan> csharp_++
15:17:26 <csharp_> 👍
15:17:47 <shulabear> csharp_++
15:17:51 <shulabear> mmorgan++
15:17:55 <shulabear> phasefx++
15:18:16 <jeffdavis> one thing about the cover uploader since we're talking about it
15:18:34 * rfrasur gets her ears all atune.
15:18:35 <jeffdavis> it will basically require multi-server environments to use an NFS share for cover images
15:19:05 <jeffdavis> IIRC if the connection to the NFS server hangs it will cause OPAC pages with cover images to fail to load
15:19:38 <jeffdavis> I haven't had a chance to reproduce this yet but wanted to flag it early
15:20:23 <phasefx> could do less real-time and more klunky stuff, but yeah
15:21:01 <jeffdavis> I don't think it should block the new feature from being committed or anything, just something to look out for once the feature goes in
15:21:23 * mmorgan also has one more comment
15:21:46 <phasefx> but the problem space should be explored already for a given installation with regards to reports
15:22:04 <mmorgan> On my test server, I had to create the directories for the cover images, so thought mention of that should go in release notes.
15:22:11 <JBoyer> Yeah, that's worth documenting since I think this is the one use of shared space that could cause patron-facing problems. (Though I don't know what happens if the existing custom covers stuff isn't repsonding)
15:23:50 <jeffdavis> anyway I'll test more and file a bug when I can
15:24:14 <phasefx> jeffdavis++
15:24:18 <mmorgan> jeffdavis++
15:24:45 <JBoyer> mmorgan, I forgot the vandelay settings had an effect here, make sure that your importer directory isn't /tmp, because that doesn't work on more recent Ubuntu and Debian releases.
15:25:13 <JBoyer> Something like /openils/var/data/vandelay is a good alternative (that would also need to be created)
15:25:47 <JBoyer> Speaking of, we should change that in the example config because /tmp doesn't work on single-server installs anymore etiehr.
15:25:48 <mmorgan> JBoyer: Good thought, but that's not the issue. Many files upload just fine.
15:25:57 <JBoyer> Ah, well.
15:26:28 <mmorgan> May have something to do with files that have been manipulated.
15:27:17 <mmorgan> But they still fit the specs re size and type
15:27:26 <JBoyer> Hmm, possibly something abut metadata tags that the perl modules don't like.
15:27:51 <JBoyer> Anyway, things to consider once this meeting is wrapped.
15:27:54 <JBoyer> #topic Announcements
15:27:58 <JBoyer> #info Next Meeting is April 12, 2022
15:28:06 <JBoyer> #endmeeting