DependencyConflict: requested: "psycopg2 >= 2.7.3.1" but found: "None"

[eugeniyost]

After upgrade opentelemetry-instrumentation-psycopg2 from 0.19b0 to 0.23b2 there was an message DependencyConflict: requested: "psycopg2 >= 2.7.3.1" but found: "None", my project dependencies do not have psycopg2, there is psycopg2-binary.

Describe your environment

• python 3.9.6
• django 3.2.5
• uwsgi 2.0.19.1
• psycopg2-binary 2.9.1
• opentelemetry-instrumentation-psycopg2 0.23b2

Steps to reproduce
install psycopg2-binary to project dependencies and opentelemetry-instrumentation-psycopg2 0.23b2, run django server using uwsgi

What is the expected behavior?
no DependencyConflict message

What is the actual behavior?
message DependencyConflict: requested: "psycopg2 >= 2.7.3.1" but found: "None"

[Dipenbhatt03]

Did you get any solution, i am trying to setup this whole distributed tracing and stuck at instrumenting psycopg2_binary.

[owais]

psycopg project recommends using psycopg2 in production instead of psycopg binary so we preferred to use that as the dependency.

https://www.psycopg.org/docs/install.html#psycopg-vs-psycopg-binary

For production use you are advised to use the source distribution.

If there are use cases where one might want to use the binary version, may be we can add both deps and update our dep resolution system treat such cases as OR so that presence of either library resolves the requirement.

[justmobilize]

We also use psycopg-binary due to how our app is built and deployed. We are trying to figure out if we need to fork or come up with another option.

Although harder for you to maintain, instead of updating your dep resolution system, you could duplicate opentelemetry-instrumentation-psycopg2 and make opentelemetry-instrumentation-psycopg2-binary (or at least for a short term solution)

[justmobilize]

Or something like this:

def get_dependency_conflicts(
    deps: Collection[str],
) -> Optional[DependencyConflict]:
    error = None
    for dep in deps:
        dep_options = dep.split("|")
        for dep_option in dep_options:
            try:
                error = None
                get_distribution(dep_option)
                break
            except VersionConflict as exc:
                error = DependencyConflict(dep, exc.dist)
            except DistributionNotFound:
                error = DependencyConflict(dep)
            except RequirementParseError as exc:
                logger.warning(
                    'error parsing dependency, reporting as a conflict: "%s" - %s',
                    dep,
                    exc,
                )
                error = DependencyConflict(dep)
        if error:
            break
    return error

[justmobilize]

@owais can I help open a pull request for one of these options?

[owais]

That would be great but please summarize your solution here first. My only concern is if we'll ever need to support the AND case in addition to OR and if we should account for that today.

[justmobilize]

@owais isn't AND already supported with:

_instruments = ("package1=1.0","package2=1.0")

in which case:

_instruments = ("package1=1.0","package2=1.0|package2-binary=1.0")

Would work with the above option.

If I'm missing something (which I may be since I'm not as familiar with all the instrumentation yet). Please let me know and I'll write up a solution

[phillipuniverse]

Here is an extremely disgusting workaround that I'm using in the meantime. Caveat - this only really works because I'm doing manual "auto-instrumentation" which you're not supposed to do. But it does work!

import opentelemetry

orig_get_dependency_conflicts = opentelemetry.instrumentation.dependencies.get_dependency_conflicts

def psycopg2_or_psycopg2_binary_dependency_conficts(
    deps: typing.Collection[str],
) -> typing.Optional[DependencyConflict]:
    if 'psycopg2>=2.7.3.1' in deps:
        conflict = orig_get_dependency_conflicts(deps)
        if conflict and not conflict.found:
            return orig_get_dependency_conflicts(['psycopg2-binary>=2.7.3.1'])
    return orig_get_dependency_conflicts(deps)

opentelemetry.instrumentation.dependencies.get_dependency_conflicts = psycopg2_or_psycopg2_binary_dependency_conficts
# This import kicks off all of the autoinstrumentation of OpenTelemetry
import opentelemetry.instrumentation.auto_instrumentation.sitecustomize as autotrace

[owais]

@justmobilize "package2=1.0|package2-binary=1.0" < this is not something supported by pip out of the box right? So you are proposing a syntax to denote OR? That could work but I feel if one string/element always represented a single package, it'd be so much easier to work with. How about this:

_instruments = Collection[Collection[Package...]...]

Inner tuples would be AND'ed while outer ones would be OR'ed. For example:

_instruments = (('a', 'b',), ('c',))

would mean the instrumentation will be applied either when both a and b are found or when c is found. So psycopg use case would look like

_instruments = (('psycopg2'), ('pyscopg-binary'),)

[justmobilize]

@owais I can do that. Just the guidance I needed. Should I follow SOP and join the Slack channel and find a OpenTelemetry buddy?

[owais]

@justmobilize no, you don't have to. You can just send the PR and discuss everything on Github. You're welcome to join the slack though :)

[MatheusGeiger]

I managed to make the instrumentation of psycopg2 work normally with psycopg2-binary.

via the flag, skip_dep_check=True makes opentelemetry-instrumentation-psycopg2 not perform dependency checks by adding the psycopg2 connection wrapper.

As internally in psycopg2 the psycopg2 and psycopg2-binary versions use the same python code(import, codebase, functions) and everything worked normally.

I know that using skip_dep_check=True can have some side effects if the library is not installed in the project but it was a way that worked for me.

more details in this discussion

[ocelotl]

@eugeniyost @justmobilize @Dipenbhatt03 does @MatheusGeiger skip_dep soultion works for you?

[robotadam]

@justmobilize is out for a while, but @mmchugh and I also worked on this issue. We haven't tested skip_dep but it looks like a good workaround. I will try to carve off time to test later this week and get back to you all.

[robotadam]

@ocelotl Sorry for the slow response, but I got around to testing this and adding skip_dep_check=True solves our problem. I'm removing our hack and we're back to using the main package. Thank you @MatheusGeiger!

[povilasv]

Another solution to this problem - theoretically could be to make a new opentelemetry-instrumentation-psycopg2-binary package, which would import code from opentelemetry-instrumentation-psycopg2 but just advertise support for "psycopg2-binary >= 2.7.3.1" instead of `"psycopg2 >= 2.7.3.1"

[bartosz-banachewicz-asi]

skip_dep_check worked for me.

[xmo-odoo]

Also fixes the issue for me when trying to trace in a development environment. Might be worth adding this to the documentation / example?

psycopg2 2.7.3.1 is more than 7 years old, even debian's oldoldstable (buster) ships 2.7.7, so I don't thing the version check is super critical for it anyway: I would assume packages which try to instrument psycopg2 need psycopg2 themselves, whether it's as psycopg2 or psycopg2-binary.

[xrmx]

That should be easy to fix, just need to do whatever the kafka-python instrumentation is doing in pyproject.toml and package.py.

[joshowen]

That should be easy to fix, just need to do whatever the kafka-python instrumentation is doing in pyproject.toml and package.py.

@xrmx I made those changes on #3186