update PRs for java dependencies

[rmuir]

Description

I added dependabot.yml in #14462

Currently it sends us pull requests for:

• github actions
• pip dependencies in dev-tools/

But nothing yet for java dependencies. I think it might be enough to rename versions.toml to libs.versions.toml to get (build-failing) pull requests? From the docs I have read, the filename is not configurable.

If the filename is not so important, it would be nice to rename it, just so that github "understands" our dependency tree and allows for features around that (such as security ones).

In order to make PRs nice, where they stand a chance to pass, it would be more work. Seems the recommended way is to integrate with actions in order to run the "post upgrade commands" and issue a commit with them. (#14506

Alternatively we could do renovatebot, but we don't have many dependencies and our needs are simple, so it would be cool if we could have dependabot fully working for us.

[rmuir]

I'm not actively working this yet, it is just for discussion. But if we can coerce the thing to work well with our project, I think it is beneficial. Better to have a failing PR, well-formatted, with release notes info, etc, to upgrade a dependency, than no PR at all? And managing completely manually?

I do think it would be a little tricky to configure all the automation necessary to create "passing by default" PRs, that dont require additional commits by developers, to be merged. It is more than just regenerating lockfile and licenses/shas, it is also regenerate tasks!

[dweiss]

I think the "gradle convention" is to keep libs.version.toml under gradle/ and I think it's what dependabot understands - and I personally hate this, without any specific objective reason... Moving it is easy but, to me, it hides the source of dependencies from its prominent top-level location. See this commit: dweiss@8b99dab

If you feel it's ok, feel free to cherry pick it to main.

[rmuir]

@dweiss mainly I am curious if we can see the new dependencies under https://github.com/apache/lucene/network/dependencies

I don't know how to test this (i think it only evals default branch). But as you can see, several capabilities such as security alerts, SBOM generation, etc would all suddenly work. Separately we can configure the dependabot for that ecosystem, if we can get it to be recognized.

Not sure where the file has to be for that, maybe the root is ok? dependabot.yml lets us specify the root directory, just not the filename to look for. For example see https://github.com/apache/lucene/blob/main/.github/dependabot.yml#L15 where we configure it for python only in dev-tools/

[dweiss]

Let me take a look on my fork.

[rmuir]

Thanks for hacking at this one, I appreciate it!

[dweiss]

Can't see the dependencies in gradle ecosystem here, perhaps yet:
https://github.com/dweiss/lucene/network/dependencies
but something did run:
https://github.com/dweiss/lucene/actions/runs/14463230772/job/40559676963#step:3:1398

The location of the libs.gradle.toml is I think fixed to gradle/, regardless of the directory setting.
https://docs.github.com/en/code-security/dependabot/ecosystems-supported-by-dependabot/supported-ecosystems-and-repositories#gradle
source:
https://github.com/dependabot/dependabot-core/blob/main/gradle/lib/dependabot/gradle/file_fetcher.rb#L24-L26

[dweiss]

I don't think the dependency graph supports gradle out of the box -

https://docs.github.com/en/code-security/supply-chain-security/understanding-your-software-supply-chain/dependency-graph-supported-package-ecosystems#supported-package-ecosystems

they recommend using an action and submitting the graph from there. Looks like this:

https://github.com/dweiss/lucene/network/dependencies?q=ecosystem%3AMaven

[dweiss]

The documentation for dependency submission action is here:
https://github.com/gradle/actions/blob/main/docs/dependency-submission.md

it is verbose. I'm not sure we need to use this though - maybe just the dependabot scan is enough?

[dweiss]

Ok, so I think that:

• adding gradle dependencies requires moving the catalog, it's a simple change and is fine,
• we get update PRs for free then
• adding dependency graph requires using an action; the result is very verbose and contains all sorts of build-time dependencies. It could probably be improved with exclusions (that this action allows) but out of the box it seems too verbose to me.

Do we want to start by moving the catalog and adding dependabot first?

[dweiss]

Here is why I think adding dependency graph will require some filtering - these reports very likely originate from gradle plugins somewhere. It's hard to even track them down:
https://github.com/dweiss/lucene/security/dependabot?q=is%3Aopen+ecosystem%3AMaven

[rmuir]

Is the catalog treated separately than the ordinary java library deps (e.g. commons-whatever)? Ideally we bring all the deps into automation, but for this issue I figured the java library deps would be more than plenty of a challenge!

[rmuir]

Here is why I think adding dependency graph will require some filtering - these reports very likely originate from gradle plugins somewhere. It's hard to even track them down:
https://github.com/dweiss/lucene/security/dependabot?q=is%3Aopen+ecosystem%3AMaven

for me that is an HTTP-404 unfortunately.

[rmuir]

I see your point: I see it from your fork at the expected place that requires no special permissions: https://github.com/dweiss/lucene/network/dependencies?q=ecosystem%3AMaven

I dont know what this "Github Dependency Graph Gradle Plugin" is doing at all. Maybe need to look more into the dependabot-core to see what it is doing. I thought I had the filenames correct.

Seems to me, this one is dragging in transitives as directs.

[rmuir]

and just my high-level suggestion: we don't want to be in that business whatsoever (managing transitive dependencies with explicit versions), or it is a bad place for update automation. this is the package manager's job to find you the one that meets all constraints.

best to just manage the top-level direct dependencies with PRs. and regenerate lockfiles with these PRs.

[dweiss]

I dont know what this "Github Dependency Graph Gradle Plugin" is doing at all

I think it's computing a dependency graph inside gradle, then submitting it to github via their API. This is kind of what we do in versions.lock but we restrict the configurations involved. It's too complicated to me. I think just the dependabot section + moving the catalog will be sufficient to get update PRs. It's simpler and should work as a start.

[dweiss]

This is I think the least invasive change:
#14495

[dweiss]

I've merged this on main. Let's see what happens.