coreforecast not compatible with IBM power chipset

[nauscj]

Description

Coreforecast isn't compatible with IBM power chipset. The other packages in the NIXTLA universe are all noarch and compatible with IBM power, but since coreforecast is not they aren't available on the chipset. We were able to install version 0.10.0 of mlforecast, version 0.2.11 of utilsforecast, and, version 1.7.3 of statsforecast. We would love to have this available in our production environment. I have been developing with mlforecast for the last year and have been trying to get it adopted as our preferred timeseries package (currently prophet), but it is unlikely if we can't get more up to date versions of the packages. Thanks for your help and all your work on these packages.

Use case

Make coreforecast noarch and available on IBM power chipset

[jmoralez]

Can you elaborate on what you mean by "isn't compatible"? We don't ship pre-compiled wheels but the package can be built from source.

[nauscj]

Since we have to build the wheel file ourselves conda can't see the coreforecast package while building the environment with a yaml. So it won't pull in the newer versions of the Nixtla packages. Additionally, it has not been able to be directly installed via pip.

We tried building a ppc64le wheel for coreforecast, and even though it was (sometimes) installable, it didn't make a difference when building the rest of the env because conda couldn't recognize it.

[jmoralez]

What happens if you specify it under the pip section of the yaml file?

[nauscj]

Because pip dependencies are executed last the conda environment tries to pull all the other packages first, but since coreforecast doesn't have a ppc64le installer (or noarch like the other packages) on conda-forge (https://anaconda.org/conda-forge/coreforecast) it keeps checking versions until it solves the env. Then it moves on to the pip section. The IBM product we use requires us to use conda as the environment solver, so we are limited in that regards.

This is the test yaml we ran...

channels:
  - anaconda
  - conda-forge
  - defaults
dependencies:
  - python=3.10.*
  - tornado
  - cryptography
  - openssl
  - pyyaml=5.*
  - pyspark=3.3.1
  - mlforecast
  - numba
  - numpy
  - pandas
  - plotly
  - pyjanitor
  - scikit-learn
  - statsforecast
  - utilsforecast
  - xgboost
  - pip
  - pip:
    - coreforecast

[jmoralez]

Ok, let me see if I can push a build to conda forge with ppc64 support.

[jmoralez]

There's now a build in conda-forge for ppc64le, please let me know if that fixes the problem.

[nauscj]

Thank you! I will have our environment manager try and build the environment today.

[nauscj]

We were able to successfully build the environment. Thanks again for your help, I really appreciate it. Just out of curiosity how are you able to make your packages chipset agnostic? We often struggle with packages not being compatible with our chipset.

[jmoralez]

Thanks for the report.

Python-only packages are noarch, so there's no problem with those, but since this package contains C++ code it needs to be compiled and that becomes arch-specific, so what we have to do is compile ahead of time for each of the platforms that we want to support and publish the package with the compiled shared library. We didn't build for ppc64le so that was the issue, but in conda it's very easy to do, we just needed to add the following line: https://github.com/conda-forge/coreforecast-feedstock/blob/276f8db948d4e98d29106d2629bb53dce8147f5a/conda-forge.yml#L11 and that built it for that arch, you can see the files here.

Closing this for now. Feel free to open another issue in statsforecast because we also have C++ code there and we don't build for ppc64le there either, so you probably got an older version.