15:00:01 <Bmagic> #startmeeting 2024-04-09 - Developer Meeting
15:00:01 <pinesol> Meeting started Tue Apr  9 15:00:01 2024 US/Eastern.  The chair is Bmagic. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:01 <pinesol> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:01 <pinesol> The meeting name has been set to '2024_04_09___developer_meeting'
15:00:09 <Bmagic> #info Agenda at https://wiki.evergreen-ils.org/doku.php?id=dev:meetings:2024-04-09
15:00:17 <Bmagic> #topic Introductions#topic Introductions
15:00:21 <Bmagic> #info Bmagic = Blake GH, MOBIUS
15:00:27 <Dyrcona> #info Dyrcona = Jason Stephenson, C/W MARS, Inc.
15:00:36 <phasefx> #info phasefx = Jason Etheridge, EOLI
15:00:45 <berick> #info berick Bill Erickson, KCLS
15:00:57 <Stompro> #info Stompro = Josh Stompro, LARL
15:01:00 <mmorgan> #info mmorgan = Michele Morgan, NOBLE
15:01:06 <sleary> #info sleary = Stephanie Leary, EOLI
15:01:56 <jeff> #info jeff = Jeff Godin, Traverse Area District Library (TADL)
15:02:01 <Bmagic> feel free to introduce yourselves throughout
15:02:11 <Bmagic> #topic Action Items from Last Meeting
15:02:19 <Bmagic> #info mmorgan will explore moving LP stats to community site and automating same
15:02:29 <mmorgan> Still looking at this. I did just want to share this sheet with some charts from the stats since I started compiling them:
15:02:35 <sandbergja> #info sandbergja = Jane Sandberg, PUL
15:02:40 <mmorgan> https://docs.google.com/spreadsheets/d/1igd02X0VjIcJrGdmcEQ34bAha8b3IHYprHkdZhcOkkE/edit?usp=sharing
15:03:12 <Bmagic> mmorgan++
15:03:21 <sandbergja> charts++
15:03:27 <sandbergja> mmorgan++
15:03:42 <sleary> mmorgan++
15:04:06 <Bmagic> I think it's the right thing
15:04:13 <JBoyer> JBoyer = Jason Boyer, EOLI
15:04:19 <JBoyer> #info JBoyer = Jason Boyer, EOLI
15:04:57 <mmorgan> Any input appreciated!
15:05:10 <Bmagic> thanks mmorgan! I suppose the hurdle we have is getting the raw data in there
15:05:31 <Bmagic> seems like we need a program (maybe we already do?) to interact with LP's API
15:06:17 <mmorgan> Dyrcona shared some python scripts which I've been looking at.
15:06:28 <mmorgan> Still coming up the curve.
15:06:39 <Bmagic> Dyrcona++
15:06:50 <mmorgan> Dyrcona++
15:06:57 <Bmagic> let us know if anyone can help!
15:06:57 <Dyrcona> Those were mostly for manipulating bug statuses, and last time I tried them they didn't work. The API has changed over the years.
15:07:19 <scottangel> #info scottangel = Scott Angel, MOBIUS
15:07:44 <mmorgan> Dyrcona: I did manage to get counts back with a few changes, so it's a start.
15:07:55 <Dyrcona> mmorgan++
15:08:00 <Bmagic> excellent
15:08:15 <Bmagic> anything else?
15:08:22 <mmorgan> Not at this time.
15:08:26 <Bmagic> #info gmcharlt - create a Git commit message type and update bug 2051946
15:08:26 <pinesol> Launchpad bug 2051946 in Evergreen "institute a Git commit message template" [Wishlist,New] https://launchpad.net/bugs/2051946 - Assigned to Galen Charlton (gmc)
15:08:42 <Bmagic> I don't suppose he's here, and it seems* like it's done at this point
15:09:15 <Bmagic> I've used the code that generates the release notes, pretty snazzy
15:10:26 <Bmagic> This bug is about committing a template, like the one described here: https://gist.github.com/lisawolderiksen/a7b99d94c92c6671181611be1641c733
15:10:28 <Bmagic> is that right?
15:11:59 <Stompro> Bmagic, I think you are correct.
15:12:11 <Bmagic> ok, we'll kick it down the road
15:12:23 <Bmagic> #action gmcharlt - create a Git commit message type and update bug 2051946
15:12:23 <pinesol> Launchpad bug 2051946 in Evergreen "institute a Git commit message template" [Wishlist,New] https://launchpad.net/bugs/2051946 - Assigned to Galen Charlton (gmc)
15:12:33 <Bmagic> #action mmorgan will explore moving LP stats to community site and automating same
15:12:50 <Bmagic> #topic Launchpad Status (as of noon Eastern)
15:13:01 <Bmagic> #info Open Bugs - 3063
15:13:02 <Bmagic> #info Pullrequests - 92
15:13:02 <Bmagic> #info Signedoff - 13
15:13:07 <Bmagic> #topic Launchpad Status since last meeting
15:13:11 <Bmagic> #info Bugs Added - 42
15:13:11 <Bmagic> #info Pullrequest tag Added - 43
15:13:11 <Bmagic> #info Signedoff tag Added - 34
15:13:11 <Bmagic> #info Fix Committed - 37
15:13:23 <Bmagic> #topic New Business - LP#1017990 - let's deprecate and/or remove the open-ils.circ.holds.create APIs!
15:14:34 <Bmagic> jeffdavis?
15:14:48 <Bmagic> anyone want to comment on the idea?
15:15:31 <Dyrcona> We usually announce the removal of something a release or two in advance to give people warning who may be using those APIs n custom apps.
15:16:09 <berick> +1 to deprecating
15:16:15 <berick> there are better apis
15:16:42 <kenstir> The Hemlock app has not called open-ils.circ.holds.create in 5+ years.
15:16:56 <Dyrcona> Deprecate for 3.13 and remove in whatever comes after?
15:17:29 <Bmagic> Dyrcona++ # yes, that sounds reasonable, considering this bug is 12+ years old
15:18:01 <mmorgan> +1
15:18:19 <Bmagic> It's already a thread on the dev list?
15:19:27 <Bmagic> I don't see it actually. Should this become an emailed announcement+blog post?
15:20:42 <Dyrcona> Yes, and it should be in the release notes for the next release as well.
15:21:05 <Dyrcona> It would be nice if we tag an API as deprecated and it would generate a warning when used.
15:21:15 <Bmagic> anyone want to volunteer for the email+blog post?
15:22:28 <Bmagic> That would be cool, Dyrcona: are you aware of an example API where we've done that. Something to crib from?
15:23:08 <berick> $logger->warn(...) ?
15:23:10 <Dyrcona> I meant "it would be nice if we [could] tag an API as deprecated." We can't right now.
15:23:28 <Dyrcona> but, yeah, adding logger->warn() would work.
15:23:38 <Bmagic> berick: that seems easy enough, though I was thinking it would be in the return payload back to the caller
15:24:07 <Dyrcona> Putting it in the payload might break things, but then 3rd parties wouldn't get the message.
15:24:15 <berick> yeah, that's all new
15:24:22 <JBoyer> Not a good idea to just drop random notes in an API reply, especially one so old we want to remove it, may as well just take it out if it's going to start confusing old clients.
15:24:47 <Bmagic> ok, payload is out, the log warning?
15:25:06 <JBoyer> Loud logging is the best way to alert sysadmins, who can hopefully track down what's calling it
15:25:37 <Bmagic> so this already-existing branch, could use that addition?
15:25:53 <Bmagic> well, no, that branch is removing
15:26:36 <Bmagic> so, a new* branch with that change for the 3.13 release, then csharp's branch merges in 3.14?
15:26:42 <eeevil> fwiw, there is the original API versioning scheme we could leverage. that's more about letting the client choose a compat level, but we can add logging there
15:27:24 <eeevil> #info eeevil == Mike Rylander, EOLI
15:29:32 <Bmagic> so, we're logging for sure, who wants to branch that?
15:30:37 <Bmagic> I can do the announcement pieces
15:31:19 <Bmagic> #action Bmagic will write announcement+blog post for deprication of open-ils.circ.title_hold.is_possible and open-ils.circ.holds.create[.override]
15:31:41 <Bmagic> I'll fix the spelling :)
15:31:56 <berick> maybe that should just go into the LP, the intermediate patching for warning
15:31:57 <JBoyer> #action JBoyer will throw a branch on lp 1017990 to loudly inform the logs something is still calling that ^
15:31:57 <pinesol> Launchpad bug 1017990 in Evergreen "Possible to bypass holds placement limits via direct API calls" [Medium,Confirmed] https://launchpad.net/bugs/1017990
15:32:06 <Bmagic> JBoyer++
15:32:26 <Bmagic> I forgot to add this
15:32:30 <Bmagic> #link https://bugs.launchpad.net/evergreen/+bug/1017990
15:32:40 <Bmagic> Shall we move on?
15:33:14 <Bmagic> #topic New Business - LP#2042158 - should we recommend disabling Postgres JIT?
15:33:21 <Bmagic> #link https://bugs.launchpad.net/evergreen/+bug/2042158
15:33:21 <pinesol> Launchpad bug 2042158 in Evergreen "Postgres 12+ needs to have JIT disabled" [Undecided,Confirmed] - Assigned to Galen Charlton (gmc)
15:33:42 <Bmagic> yall are quiet today :)
15:35:01 <Bmagic> gmcharlt isn't here, it's assigned to him. I think he was going to explore making Evergreen JIT-compatible?
15:35:13 <sandbergja> I was curious: what about the JIT causes these performance issues?  Do we have any postgres docs saying what we would need to change?
15:35:52 <Bmagic> I found it rear it's head when saving a bib record in the staff client
15:36:46 <Dyrcona> Bmagic: You sure that's the JIT? Saving a bib can sometimes be slow because of all the triggers.
15:37:23 <Bmagic> I think* it was saving a bib record, which in-turn queried reporter.old_super_simple_record - which is what's written on the bug report
15:37:32 <eeevil> there are certain things that JIT wants to do that we have already poured a lot of brain power into manually tuning
15:37:53 <Bmagic> that makes sense
15:38:43 <eeevil> it's not that EG is incompatible with PG's JIT, it's that (just like autovac and stats targets, and lots of other stuff, tbh) it requires config that is tuned to your particular size and shape of data
15:39:18 <eeevil> a small EG instance may benefit a BUNCH from JIT, and reporting might as well (I've tested neither, but I have suspicions)
15:40:07 <sandbergja> thanks, eeevil, that's helpful info to have
15:40:19 <eeevil> but(!) unless someone has the tuits to learn the feature well, and analyze our queries in that context, I agree that we should recommend disabling JIT for now
15:40:31 <Bmagic> eeevil++
15:41:19 <eeevil> I'd actually propose we investigate (as a community, as broadly as we can manage) cross-column stats targets before jumping into JIT
15:41:41 <Dyrcona> I'm OK with that, but I don't think I've every encountered an issue caused by JIT. Pg 16 (even unoptimized) has pretty much always been faster than Pg 10 for me. BUT! I've not run it in produciton, just in testing on a production-sized database with a production-sized server.
15:41:56 <eeevil> (not to derail the JIT-specific discussion, but as a query tuning tool I think that will pay off for more folks)
15:42:43 <eeevil> Dyrcona: I haven't seen JIT-specific issues either, but I think we can be fairly sure that plan stability is /higher/ without JIT (until we learn to tune it well)
15:43:01 <Bmagic> Disabling it fixed our problem on a test-then-production instance
15:43:29 <Bmagic> pretty sure it manifested as a time-out when clicking on save in the MARC editor
15:43:43 <Dyrcona> So, we'll recommend disabling JIT in the installation instructions, then.
15:43:48 <eeevil> which is to say, yes, some folks will lose out on some optimizations they get "for free" with JIT on, but things are much less likely to go sideways in ways some have seen
15:45:28 <Bmagic> I understand that PG is sensitive to the data within, and the configuration tweaks need to match. I made tweaks to the best of my abilities to match the machine's memory compared to database size. shared_buffers, default_statistics_target, work_mem, wal_buffers etc
15:45:58 <Dyrcona> Well, tuning PostgreSQL still feels too much like voodoo to me. :)
15:46:32 <Bmagic> eeevil: maybe a different bug for cross-column stats targets?
15:46:58 <eeevil> Bmagic: oh, yes, definitely. that should not go on the JIT bug
15:47:03 <Bmagic> (we do need to move on, down to 13 minutes)
15:47:11 <Bmagic> want an action item?
15:47:34 <eeevil> want, and "have any time at all for" are non-overlapping sets, today
15:47:43 <Bmagic> #action eeevil will open a bug for cross-column stats targets
15:47:47 <Bmagic> in your face
15:47:57 <eeevil> I've been volun-told!
15:47:58 <Bmagic> :)
15:48:34 <Bmagic> I think we can leave the JIT discussion there, PG 12+ is still experimental
15:48:52 <Bmagic> I'll add it to next month's topics too
15:49:00 <Bmagic> #topic New Business - Redis licensing change / discussion of potential replacement
15:49:17 <Dyrcona> Looks like the distros are going with ValKey.
15:49:43 <berick> yeah
15:49:46 <jeff> based on LF's support, yeah.
15:49:49 <jeff> works for me!
15:49:57 <Bmagic> ValKey++
15:49:59 <Bmagic> berick++
15:50:05 <berick> it'll be a while before valkey is packagable .. tons of activity on there now
15:50:09 <Bmagic> redis--
15:50:10 <berick> to rebrand, etc.
15:50:52 <berick> i mean it's usable, but not something you want to create a lot of documentation around just yet
15:51:05 <Bmagic> any clue when* ?
15:51:09 <berick> not I
15:52:04 <Bmagic> 8 minutes..... anything else on this one?
15:52:05 <berick> i guess question is, continue as-is with Redis, then switch later, or pause the whole shebang until Valkey is ready enough for the community
15:52:13 <berick> but we can discuss more at the Hatckfest
15:52:19 <jeff> with current "Evergreen works on these distros" packaging redis (a version before the license change), is there any reason/value in installing ValKey on those systems? Use distro-supplied redis packages until distro-supplied ValKey packages are the norm?
15:53:08 <Bmagic> we've not merged the OpenSRF branch, so, we can wait right?
15:53:21 <berick> jeff: yeah, i don't see any need to rush the valkey.  it's the same code.
15:53:38 <jeff> if there's reason to install ValKey (a la pgdg apt repos like with Postgres), then by all means... but if not, it's... yeah, just a name in this scenario.
15:53:42 <berick> rather, any need to run screaming from Redis just yet
15:53:55 <Bmagic> or, jeff, are you saying, we merge the branch with the older redis sooner, rather than waiting for valkey?
15:54:47 <berick> merging sooner would be my pref, but that's probably no surprise
15:55:01 <jeff> close. i was asking if there was reason to wait for valkey to stabilize and be widely available in our favorite linux distros by default. sounds like there's no need to wait.
15:55:04 <Bmagic> to be continued at hackfest
15:55:12 <berick> Bmagic: how's it going, btw, w/ Redis?
15:55:35 <Bmagic> It's running so fine, it's just working. I don't think about it
15:55:58 * Dyrcona is going to try Redis on Ubuntu 24.04 to see if it affects the buffer overflow I get when starting OpenSRF services.
15:56:03 <Bmagic> but yes, we have it on a small production instance
15:56:04 <berick> awesome Bmagic++
15:56:14 <Bmagic> #topic New Business - Release branch proposals – anything we can move forward with?
15:56:20 <Bmagic> sorry we're probably going to go over time
15:56:24 <Bmagic> #link http://list.evergreen-ils.org/pipermail/evergreen-dev/2024-March/000777.html
15:56:53 <sandbergja> Thanks for all who have discussed this on the mailing list so far
15:57:09 <sandbergja> Just wondering if there is anything there that is actionable
15:57:57 <sandbergja> in the interest of reducing the number of things we have to think about while building releases (which is currently too many for me personally, I always find that I create the release tag branch at the wrong time in the process or some other mistake)
15:58:33 <Bmagic> tagging the branch rather than branching the tag
15:58:52 <Dyrcona> It would be nice to just type 'make release' and not have to worry about anything else. :)
15:59:06 <sandbergja> 100%
15:59:32 <Bmagic> and our PG version-upgrade stuff would auto-figure itself
16:00:24 <eeevil> I just want to register my dislike of "stamp version commit -> cut release -> UNstamp version commit" ... but other than that (if we can avoid it) it's all just a checklist
16:00:28 <Dyrcona> Bmagic: It's mostly automatic now, but there might be a few steps required before running 'make release.'
16:01:07 <sandbergja> eeevil: does your objection still apply to what Dyrcona proposed, with the stamping it "+dev" between releases?
16:01:09 <Bmagic> eeevil: the tag would presumabely remain on the commit right?
16:02:30 <eeevil> sandbergja: I'm less against that, but anything that keeps main (or one of the major version branches) effectively locked gets the side-eye from me
16:03:39 <Dyrcona> eeevil: What do you mean by "effectively locked?"
16:04:35 <sandbergja> oh, like locked to a specific version?
16:05:16 <eeevil> well, today we say "ok, what's in there now is 3.13.0, make a branch for that" -- at which point commits for 3.13.1 can start flowing into th 3.13 branch -- "and fill it with as many release-related commits as are needed"
16:05:54 <eeevil> the only "locking" is the exact branch point, and that's not locked, we can branch from a commit 3 back from the tip
16:06:49 <eeevil> but, if we put all the release related commits into the overall version branch then nobody but the release team can touch that branch until the all-clear is sent out
16:07:28 <eeevil> it's effectively locked because we can't have non-release-related commits going in
16:07:38 <Dyrcona> That's generally how we do it now anyway.
16:08:10 <Dyrcona> Someone almost always send an email: We're working on relases, don't touch branches X, Y, Z.
16:08:26 <eeevil> I know I haven't been RM in quite a while, but ... why do we do that?
16:09:11 <Dyrcona> Because building releases is WAY more time consuming than it used to be and involves 4 to 6 people generally. Even point releases.
16:09:24 <sandbergja> what Dyrcona said
16:09:32 <sandbergja> we may be getting a bit off topic, as well
16:10:29 <eeevil> (I could very well be misremembering, but I recall my process being: 1) release branch 2) version update in there 3) upgrade script in there 4) release notes in there 5) side-port upgrade script and release notes to major version and main)
16:10:35 <sandbergja> If I put together a branch that has some automation of 1) git tags, 2) keeping all commits on the rel_* and main branches, 3) stamping the +dev when we are not in a release, and 4) removes Changelog, would anyone object?
16:11:34 <Dyrcona> eeevil: We get translations from 2 sources, do release notes for bug fixes, backport release notes, etc.
16:12:02 <Dyrcona> more like forward port release notes and db upgrades.
16:12:39 <Dyrcona> sandbergja: I'll look at the branch if you put one together.
16:12:50 <sandbergja> Dyrcona++
16:12:55 <Bmagic> Dyrcona++
16:13:05 <Bmagic> sorry to keep yall late
16:13:10 <eeevil> sandbergja: I certainly won't object to a branch to look at! ;)
16:13:11 <Bmagic> #topic Announcements
16:13:16 <sandbergja> Bmagic++
16:13:17 <Bmagic> #info Next Meeting is Tuesday, May 14th 2024
16:13:22 <Bmagic> #endmeeting