15:00:01 #startmeeting 2024-04-09 - Developer Meeting 15:00:01 Meeting started Tue Apr 9 15:00:01 2024 US/Eastern. The chair is Bmagic. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:01 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:01 The meeting name has been set to '2024_04_09___developer_meeting' 15:00:09 #info Agenda at https://wiki.evergreen-ils.org/doku.php?id=dev:meetings:2024-04-09 15:00:17 #topic Introductions#topic Introductions 15:00:21 #info Bmagic = Blake GH, MOBIUS 15:00:27 #info Dyrcona = Jason Stephenson, C/W MARS, Inc. 15:00:36 #info phasefx = Jason Etheridge, EOLI 15:00:45 #info berick Bill Erickson, KCLS 15:00:57 #info Stompro = Josh Stompro, LARL 15:01:00 #info mmorgan = Michele Morgan, NOBLE 15:01:06 #info sleary = Stephanie Leary, EOLI 15:01:56 #info jeff = Jeff Godin, Traverse Area District Library (TADL) 15:02:01 feel free to introduce yourselves throughout 15:02:11 #topic Action Items from Last Meeting 15:02:19 #info mmorgan will explore moving LP stats to community site and automating same 15:02:29 Still looking at this. I did just want to share this sheet with some charts from the stats since I started compiling them: 15:02:35 #info sandbergja = Jane Sandberg, PUL 15:02:40 https://docs.google.com/spreadsheets/d/1igd02X0VjIcJrGdmcEQ34bAha8b3IHYprHkdZhcOkkE/edit?usp=sharing 15:03:12 mmorgan++ 15:03:21 charts++ 15:03:27 mmorgan++ 15:03:42 mmorgan++ 15:04:06 I think it's the right thing 15:04:13 JBoyer = Jason Boyer, EOLI 15:04:19 #info JBoyer = Jason Boyer, EOLI 15:04:57 Any input appreciated! 15:05:10 thanks mmorgan! I suppose the hurdle we have is getting the raw data in there 15:05:31 seems like we need a program (maybe we already do?) to interact with LP's API 15:06:17 Dyrcona shared some python scripts which I've been looking at. 15:06:28 Still coming up the curve. 15:06:39 Dyrcona++ 15:06:50 Dyrcona++ 15:06:57 let us know if anyone can help! 15:06:57 Those were mostly for manipulating bug statuses, and last time I tried them they didn't work. The API has changed over the years. 15:07:19 #info scottangel = Scott Angel, MOBIUS 15:07:44 Dyrcona: I did manage to get counts back with a few changes, so it's a start. 15:07:55 mmorgan++ 15:08:00 excellent 15:08:15 anything else? 15:08:22 Not at this time. 15:08:26 #info gmcharlt - create a Git commit message type and update bug 2051946 15:08:26 Launchpad bug 2051946 in Evergreen "institute a Git commit message template" [Wishlist,New] https://launchpad.net/bugs/2051946 - Assigned to Galen Charlton (gmc) 15:08:42 I don't suppose he's here, and it seems* like it's done at this point 15:09:15 I've used the code that generates the release notes, pretty snazzy 15:10:26 This bug is about committing a template, like the one described here: https://gist.github.com/lisawolderiksen/a7b99d94c92c6671181611be1641c733 15:10:28 is that right? 15:11:59 Bmagic, I think you are correct. 15:12:11 ok, we'll kick it down the road 15:12:23 #action gmcharlt - create a Git commit message type and update bug 2051946 15:12:23 Launchpad bug 2051946 in Evergreen "institute a Git commit message template" [Wishlist,New] https://launchpad.net/bugs/2051946 - Assigned to Galen Charlton (gmc) 15:12:33 #action mmorgan will explore moving LP stats to community site and automating same 15:12:50 #topic Launchpad Status (as of noon Eastern) 15:13:01 #info Open Bugs - 3063 15:13:02 #info Pullrequests - 92 15:13:02 #info Signedoff - 13 15:13:07 #topic Launchpad Status since last meeting 15:13:11 #info Bugs Added - 42 15:13:11 #info Pullrequest tag Added - 43 15:13:11 #info Signedoff tag Added - 34 15:13:11 #info Fix Committed - 37 15:13:23 #topic New Business - LP#1017990 - let's deprecate and/or remove the open-ils.circ.holds.create APIs! 15:14:34 jeffdavis? 15:14:48 anyone want to comment on the idea? 15:15:31 We usually announce the removal of something a release or two in advance to give people warning who may be using those APIs n custom apps. 15:16:09 +1 to deprecating 15:16:15 there are better apis 15:16:42 The Hemlock app has not called open-ils.circ.holds.create in 5+ years. 15:16:56 Deprecate for 3.13 and remove in whatever comes after? 15:17:29 Dyrcona++ # yes, that sounds reasonable, considering this bug is 12+ years old 15:18:01 +1 15:18:19 It's already a thread on the dev list? 15:19:27 I don't see it actually. Should this become an emailed announcement+blog post? 15:20:42 Yes, and it should be in the release notes for the next release as well. 15:21:05 It would be nice if we tag an API as deprecated and it would generate a warning when used. 15:21:15 anyone want to volunteer for the email+blog post? 15:22:28 That would be cool, Dyrcona: are you aware of an example API where we've done that. Something to crib from? 15:23:08 $logger->warn(...) ? 15:23:10 I meant "it would be nice if we [could] tag an API as deprecated." We can't right now. 15:23:28 but, yeah, adding logger->warn() would work. 15:23:38 berick: that seems easy enough, though I was thinking it would be in the return payload back to the caller 15:24:07 Putting it in the payload might break things, but then 3rd parties wouldn't get the message. 15:24:15 yeah, that's all new 15:24:22 Not a good idea to just drop random notes in an API reply, especially one so old we want to remove it, may as well just take it out if it's going to start confusing old clients. 15:24:47 ok, payload is out, the log warning? 15:25:06 Loud logging is the best way to alert sysadmins, who can hopefully track down what's calling it 15:25:37 so this already-existing branch, could use that addition? 15:25:53 well, no, that branch is removing 15:26:36 so, a new* branch with that change for the 3.13 release, then csharp's branch merges in 3.14? 15:26:42 fwiw, there is the original API versioning scheme we could leverage. that's more about letting the client choose a compat level, but we can add logging there 15:27:24 #info eeevil == Mike Rylander, EOLI 15:29:32 so, we're logging for sure, who wants to branch that? 15:30:37 I can do the announcement pieces 15:31:19 #action Bmagic will write announcement+blog post for deprication of open-ils.circ.title_hold.is_possible and open-ils.circ.holds.create[.override] 15:31:41 I'll fix the spelling :) 15:31:56 maybe that should just go into the LP, the intermediate patching for warning 15:31:57 #action JBoyer will throw a branch on lp 1017990 to loudly inform the logs something is still calling that ^ 15:31:57 Launchpad bug 1017990 in Evergreen "Possible to bypass holds placement limits via direct API calls" [Medium,Confirmed] https://launchpad.net/bugs/1017990 15:32:06 JBoyer++ 15:32:26 I forgot to add this 15:32:30 #link https://bugs.launchpad.net/evergreen/+bug/1017990 15:32:40 Shall we move on? 15:33:14 #topic New Business - LP#2042158 - should we recommend disabling Postgres JIT? 15:33:21 #link https://bugs.launchpad.net/evergreen/+bug/2042158 15:33:21 Launchpad bug 2042158 in Evergreen "Postgres 12+ needs to have JIT disabled" [Undecided,Confirmed] - Assigned to Galen Charlton (gmc) 15:33:42 yall are quiet today :) 15:35:01 gmcharlt isn't here, it's assigned to him. I think he was going to explore making Evergreen JIT-compatible? 15:35:13 I was curious: what about the JIT causes these performance issues? Do we have any postgres docs saying what we would need to change? 15:35:52 I found it rear it's head when saving a bib record in the staff client 15:36:46 Bmagic: You sure that's the JIT? Saving a bib can sometimes be slow because of all the triggers. 15:37:23 I think* it was saving a bib record, which in-turn queried reporter.old_super_simple_record - which is what's written on the bug report 15:37:32 there are certain things that JIT wants to do that we have already poured a lot of brain power into manually tuning 15:37:53 that makes sense 15:38:43 it's not that EG is incompatible with PG's JIT, it's that (just like autovac and stats targets, and lots of other stuff, tbh) it requires config that is tuned to your particular size and shape of data 15:39:18 a small EG instance may benefit a BUNCH from JIT, and reporting might as well (I've tested neither, but I have suspicions) 15:40:07 thanks, eeevil, that's helpful info to have 15:40:19 but(!) unless someone has the tuits to learn the feature well, and analyze our queries in that context, I agree that we should recommend disabling JIT for now 15:40:31 eeevil++ 15:41:19 I'd actually propose we investigate (as a community, as broadly as we can manage) cross-column stats targets before jumping into JIT 15:41:41 I'm OK with that, but I don't think I've every encountered an issue caused by JIT. Pg 16 (even unoptimized) has pretty much always been faster than Pg 10 for me. BUT! I've not run it in produciton, just in testing on a production-sized database with a production-sized server. 15:41:56 (not to derail the JIT-specific discussion, but as a query tuning tool I think that will pay off for more folks) 15:42:43 Dyrcona: I haven't seen JIT-specific issues either, but I think we can be fairly sure that plan stability is /higher/ without JIT (until we learn to tune it well) 15:43:01 Disabling it fixed our problem on a test-then-production instance 15:43:29 pretty sure it manifested as a time-out when clicking on save in the MARC editor 15:43:43 So, we'll recommend disabling JIT in the installation instructions, then. 15:43:48 which is to say, yes, some folks will lose out on some optimizations they get "for free" with JIT on, but things are much less likely to go sideways in ways some have seen 15:45:28 I understand that PG is sensitive to the data within, and the configuration tweaks need to match. I made tweaks to the best of my abilities to match the machine's memory compared to database size. shared_buffers, default_statistics_target, work_mem, wal_buffers etc 15:45:58 Well, tuning PostgreSQL still feels too much like voodoo to me. :) 15:46:32 eeevil: maybe a different bug for cross-column stats targets? 15:46:58 Bmagic: oh, yes, definitely. that should not go on the JIT bug 15:47:03 (we do need to move on, down to 13 minutes) 15:47:11 want an action item? 15:47:34 want, and "have any time at all for" are non-overlapping sets, today 15:47:43 #action eeevil will open a bug for cross-column stats targets 15:47:47 in your face 15:47:57 I've been volun-told! 15:47:58 :) 15:48:34 I think we can leave the JIT discussion there, PG 12+ is still experimental 15:48:52 I'll add it to next month's topics too 15:49:00 #topic New Business - Redis licensing change / discussion of potential replacement 15:49:17 Looks like the distros are going with ValKey. 15:49:43 yeah 15:49:46 based on LF's support, yeah. 15:49:49 works for me! 15:49:57 ValKey++ 15:49:59 berick++ 15:50:05 it'll be a while before valkey is packagable .. tons of activity on there now 15:50:09 redis-- 15:50:10 to rebrand, etc. 15:50:52 i mean it's usable, but not something you want to create a lot of documentation around just yet 15:51:05 any clue when* ? 15:51:09 not I 15:52:04 8 minutes..... anything else on this one? 15:52:05 i guess question is, continue as-is with Redis, then switch later, or pause the whole shebang until Valkey is ready enough for the community 15:52:13 but we can discuss more at the Hatckfest 15:52:19 with current "Evergreen works on these distros" packaging redis (a version before the license change), is there any reason/value in installing ValKey on those systems? Use distro-supplied redis packages until distro-supplied ValKey packages are the norm? 15:53:08 we've not merged the OpenSRF branch, so, we can wait right? 15:53:21 jeff: yeah, i don't see any need to rush the valkey. it's the same code. 15:53:38 if there's reason to install ValKey (a la pgdg apt repos like with Postgres), then by all means... but if not, it's... yeah, just a name in this scenario. 15:53:42 rather, any need to run screaming from Redis just yet 15:53:55 or, jeff, are you saying, we merge the branch with the older redis sooner, rather than waiting for valkey? 15:54:47 merging sooner would be my pref, but that's probably no surprise 15:55:01 close. i was asking if there was reason to wait for valkey to stabilize and be widely available in our favorite linux distros by default. sounds like there's no need to wait. 15:55:04 to be continued at hackfest 15:55:12 Bmagic: how's it going, btw, w/ Redis? 15:55:35 It's running so fine, it's just working. I don't think about it 15:55:58 * Dyrcona is going to try Redis on Ubuntu 24.04 to see if it affects the buffer overflow I get when starting OpenSRF services. 15:56:03 but yes, we have it on a small production instance 15:56:04 awesome Bmagic++ 15:56:14 #topic New Business - Release branch proposals – anything we can move forward with? 15:56:20 sorry we're probably going to go over time 15:56:24 #link http://list.evergreen-ils.org/pipermail/evergreen-dev/2024-March/000777.html 15:56:53 Thanks for all who have discussed this on the mailing list so far 15:57:09 Just wondering if there is anything there that is actionable 15:57:57 in the interest of reducing the number of things we have to think about while building releases (which is currently too many for me personally, I always find that I create the release tag branch at the wrong time in the process or some other mistake) 15:58:33 tagging the branch rather than branching the tag 15:58:52 It would be nice to just type 'make release' and not have to worry about anything else. :) 15:59:06 100% 15:59:32 and our PG version-upgrade stuff would auto-figure itself 16:00:24 I just want to register my dislike of "stamp version commit -> cut release -> UNstamp version commit" ... but other than that (if we can avoid it) it's all just a checklist 16:00:28 Bmagic: It's mostly automatic now, but there might be a few steps required before running 'make release.' 16:01:07 eeevil: does your objection still apply to what Dyrcona proposed, with the stamping it "+dev" between releases? 16:01:09 eeevil: the tag would presumabely remain on the commit right? 16:02:30 sandbergja: I'm less against that, but anything that keeps main (or one of the major version branches) effectively locked gets the side-eye from me 16:03:39 eeevil: What do you mean by "effectively locked?" 16:04:35 oh, like locked to a specific version? 16:05:16 well, today we say "ok, what's in there now is 3.13.0, make a branch for that" -- at which point commits for 3.13.1 can start flowing into th 3.13 branch -- "and fill it with as many release-related commits as are needed" 16:05:54 the only "locking" is the exact branch point, and that's not locked, we can branch from a commit 3 back from the tip 16:06:49 but, if we put all the release related commits into the overall version branch then nobody but the release team can touch that branch until the all-clear is sent out 16:07:28 it's effectively locked because we can't have non-release-related commits going in 16:07:38 That's generally how we do it now anyway. 16:08:10 Someone almost always send an email: We're working on relases, don't touch branches X, Y, Z. 16:08:26 I know I haven't been RM in quite a while, but ... why do we do that? 16:09:11 Because building releases is WAY more time consuming than it used to be and involves 4 to 6 people generally. Even point releases. 16:09:24 what Dyrcona said 16:09:32 we may be getting a bit off topic, as well 16:10:29 (I could very well be misremembering, but I recall my process being: 1) release branch 2) version update in there 3) upgrade script in there 4) release notes in there 5) side-port upgrade script and release notes to major version and main) 16:10:35 If I put together a branch that has some automation of 1) git tags, 2) keeping all commits on the rel_* and main branches, 3) stamping the +dev when we are not in a release, and 4) removes Changelog, would anyone object? 16:11:34 eeevil: We get translations from 2 sources, do release notes for bug fixes, backport release notes, etc. 16:12:02 more like forward port release notes and db upgrades. 16:12:39 sandbergja: I'll look at the branch if you put one together. 16:12:50 Dyrcona++ 16:12:55 Dyrcona++ 16:13:05 sorry to keep yall late 16:13:10 sandbergja: I certainly won't object to a branch to look at! ;) 16:13:11 #topic Announcements 16:13:16 Bmagic++ 16:13:17 #info Next Meeting is Tuesday, May 14th 2024 16:13:22 #endmeeting