LFS Endpoint not being set (Bitbucket OAuth) causing LFS pull to fail

[frankvHoof93]

Jenkins and plugins versions report

Environment

Jenkins: 2.332.2
OS: Linux - 5.10.16.3-microsoft-standard-WSL2
---
ace-editor:1.1
ant:475.vf34069fef73c
antisamy-markup-formatter:2.7
apache-httpcomponents-client-4-api:4.5.13-1.0
atlassian-bitbucket-server-integration:3.2.1
authentication-tokens:1.4
bitbucket:223.vd12f2bca5430
bitbucket-oauth:0.12
bitbucket-pullrequest-builder:1.5.0
bitbucket-pullrequests-filter:0.1.0
bitbucket-push-and-pull-request:2.8.1
blueocean:1.25.3
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.25.3
blueocean-commons:1.25.3
blueocean-config:1.25.3
blueocean-core-js:1.25.3
blueocean-dashboard:1.25.3
blueocean-display-url:2.4.1
blueocean-events:1.25.3
blueocean-git-pipeline:1.25.3
blueocean-github-pipeline:1.25.3
blueocean-i18n:1.25.3
blueocean-jira:1.25.3
blueocean-jwt:1.25.3
blueocean-personalization:1.25.3
blueocean-pipeline-api-impl:1.25.3
blueocean-pipeline-editor:1.25.3
blueocean-pipeline-scm-api:1.25.3
blueocean-rest:1.25.3
blueocean-rest-impl:1.25.3
blueocean-web:1.25.3
bootstrap4-api:4.6.0-4
bootstrap5-api:5.1.3-6
bouncycastle-api:2.26
branch-api:2.1046.v0ca_37783ecc5
build-timeout:1.20
caffeine-api:2.9.3-65.v6a_47d0f4d1fe
checks-api:1.7.3
cloudbees-bitbucket-branch-source:765.v5a_2d6a_23c01d
cloudbees-folder:6.714.v79e858ef76a_2
command-launcher:81.v9c2cb_cb_db_392
config-file-provider:3.9.0
credentials:1087.1089.v2f1b_9a_b_040e4
credentials-binding:523.vd859a_4b_122e6
discord-notifier:206.vee9f4569ee63
display-url-api:2.3.6
docker-build-step:2.8
docker-commons:1.19
docker-compose-build-step:1.0
docker-java-api:3.2.13-37.vf3411c9828b9
docker-plugin:1.2.9
docker-slaves:1.0.7
docker-workflow:1.28
durable-task:496.va67c6f9eefa7
echarts-api:5.3.2-1
email-ext:2.88
external-monitor-job:191.v363d0d1efdf8
favorite:2.4.1
font-awesome-api:6.0.0-1
git:4.11.1
git-client:3.11.0
git-server:1.10
github:1.34.3
github-api:1.303-400.v35c2d8258028
github-branch-source:1598.v91207e9f9b_4a_
google-oauth-plugin:1.0.6
google-play-android-publisher:4.2
gradle:1.38
handlebars:3.0.8
handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953
htmlpublisher:1.30
jackson2-api:2.13.2.20220328-273.v11d70a_b_a_1a_52
javadoc:217.v905b_86277a_2a_
javax-activation-api:1.2.0-3
javax-mail-api:1.6.2-6
jaxb:2.3.0.1
jdk-tool:1.5
jenkins-design-language:1.25.3
jira:3.7.1
jjwt-api:0.11.2-71.v2722b_b_06a_2a_f
jnr-posix-api:3.1.7-3
job-dsl:1.79
jquery:1.12.4-1
jquery-detached:1.2.1
jquery3-api:3.6.0-3
jsch:0.1.55.2
junit:1.60
ldap:2.9
locale:144.v1a_998824ddb_3
lockable-resources:2.14
mailer:414.vcc4c33714601
mapdb-api:1.0.9.0
matrix-auth:3.1.1
matrix-project:771.v574584b_39e60
maven-plugin:3.18
mercurial:2.16
momentjs:1.1.1
oauth-credentials:0.5
okhttp-api:4.9.3-105.vb96869f8ac3a
pam-auth:1.8
pipeline-build-step:2.18
pipeline-github-lib:36.v4c01db_ca_ed16
pipeline-graph-analysis:195.v5812d95a_a_2f9
pipeline-input-step:448.v37cea_9a_10a_70
pipeline-milestone-step:101.vd572fef9d926
pipeline-model-api:2.2077.vc78ec45162f1
pipeline-model-definition:2.2077.vc78ec45162f1
pipeline-model-extensions:2.2077.vc78ec45162f1
pipeline-rest-api:2.24
pipeline-stage-step:293.v200037eefcd5
pipeline-stage-tags-metadata:2.2077.vc78ec45162f1
pipeline-stage-view:2.24
pipeline-utility-steps:2.12.0
plain-credentials:1.8
plugin-util-api:2.16.0
popper-api:1.16.1-3
popper2-api:2.11.5-1
preSCMbuildstep:0.3
pubsub-light:1.16
resource-disposer:0.19
run-condition:1.5
scm-api:608.vfa_f971c5a_a_e9
script-security:1158.v7c1b_73a_69a_08
snakeyaml-api:1.30.1
sonar:2.14
sse-gateway:1.25
ssh-credentials:277.v95c2fec1c047
ssh-slaves:1.814.vc82988f54b_10
sshd:3.228.v4c9f9e652c86
structs:318.va_f3ccb_729b_71
subversion:2.15.4
timestamper:1.17
token-macro:293.v283932a_0a_b_49
trilead-api:1.57.v6e90e07157e1
unity3d-plugin:1.3
variant:1.4
windows-slaves:1.8
workflow-aggregator:2.7
workflow-api:1144.v61c3180fa_03f
workflow-basic-steps:948.v2c72a_091b_b_68
workflow-cps:2689.v434009a_31b_f1
workflow-cps-global-lib:570.v21311f4951f8
workflow-durable-task-step:1130.v8fd69d0b_8857
workflow-job:1180.v04c4e75dce43
workflow-multibranch:712.vc169a_1387405
workflow-scm-step:399.v9b_8f4da_65061
workflow-step-api:625.vd896b_f445a_f8
workflow-support:819.v37d707a_71d9b_
ws-cleanup:0.42

What Operating System are you using (both controller, and any agents involved in the problem)?

Running in Docker.
Jenkins-Master: jenkinsci/blueocean:latest
Node doing the Checkout: jenkins/agent:latest (custom dockerfile used to update LFS to latest version)

Versions on Node:
git version 2.30.2
git-lfs/3.1.4 (GitHub; linux amd64; go 1.17.8)

Reproduction steps

1. Added a multi-branch pipeline using Bitbucket OAuth (Repository discovered via Credentials & Owner)
2. Attempted to do a 'checkout scm'-step in the first stage of my pipeline
3. Checkout fails if LFS-files are present in the repository

Expected Results

LFS files are pulled from the relevant LFS-repository for the Bitbucket-Repo set for the jenkins-project

Actual Results

Pulling the files fails with the following error:

batch request: missing protocol: "<unknown>"
Failed to fetch some objects from '<unknown>'

Anything else?

It looks like the 'endpoint' is not being set in the LFS-environment
I have tried pulling with --skip-smudge before the checkout scm and doing a manual git lfs pull, with the same result.
Tried both with & without the 'Git LFS pull after checkout'-behaviour in the project-configuration (on jenkins). The result is always the same: Missing protocol: "unknown"

Output of logging-attempt below (tokens & identifiers replaced):

+ git lfs install --skip-smudge
Git LFS initialized.

+ git lfs env

(Skipped copying empty config, this is pre-checkout)

Check out from version control:
The recommended git tool is: NONE
using credential Bitbucket_OAuth
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository [https://x-token-auth:{**TOKEN**}@bitbucket.org/**USER**/**REPO**.git](https://x-token-auth:**TOKEN**@bitbucket.org/**USER**/**REPO**.git)
 > git init /home/jenkins/agent/workspace/BuildTest_master # timeout=10
Fetching upstream changes from https://x-token-auth@bitbucket.org/**USER**/**REPO**.git
 > git --version # timeout=10
 > git --version # 'git version 2.30.2'
using GIT_ASKPASS to set credentials OAuth-Credentials for Bitbucket Repositories
 > git fetch --no-tags --force --progress -- [https://x-token-auth:{**TOKEN**}@bitbucket.org/**USER**/**REPO**.git](https://x-token-auth:**TOKEN**@bitbucket.org/**USER**/**REPO**.git) +refs/heads/master:refs/remotes/origin/master # timeout=10
Avoid second fetch
Checking out Revision ec926cd8e6735a2451a44354d052bbb97efb7713 (master)
 > git config remote.origin.url [https://x-token-auth:{**TOKEN**}@bitbucket.org/**USER**/**REPO**.git](https://x-token-auth:**TOKEN**@bitbucket.org/**USER**/**REPO**.git) # timeout=10
 > git config --add remote.origin.fetch +refs/heads/master:refs/remotes/origin/master # timeout=10
 > git config core.sparsecheckout # timeout=10
 > git checkout -f ec926cd8e6735a2451a44354d052bbb97efb7713 # timeout=10
Commit message: "no message"
First time build. Skipping changelog.
[Bitbucket] Notifying commit build result
Can not determine Jenkins root URL or Jenkins URL is not a valid URL regarding Bitbucket API. Commit status notifications are disabled until a root URL is configured in Jenkins global configuration. 
IllegalStateException: Jenkins URL cannot start with http://localhost/

I believe the issue is here.. Shouldn't "Endpoint" be filled at this point?

+ git lfs env
git-lfs/3.1.4 (GitHub; linux amd64; go 1.17.8)
git version 2.30.2
Endpoint=<unknown> (auth=none)
LocalWorkingDir=/home/jenkins/agent/workspace/BuildTest_master
LocalGitDir=/home/jenkins/agent/workspace/BuildTest_master/.git
LocalGitStorageDir=/home/jenkins/agent/workspace/BuildTest_master/.git
LocalMediaDir=/home/jenkins/agent/workspace/BuildTest_master/.git/lfs/objects
LocalReferenceDirs=
TempDir=/home/jenkins/agent/workspace/BuildTest_master/.git/lfs/tmp
ConcurrentTransfers=8
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=/home/jenkins/agent/workspace/BuildTest_master/.git/lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic,lfs-standalone-file,ssh
UploadTransfers=basic,lfs-standalone-file,ssh
GIT_EXEC_PATH=/usr/lib/git-core
GIT_LFS_ENABLED=true
git config filter.lfs.process = "git-lfs filter-process --skip"
git config filter.lfs.smudge = "git-lfs smudge --skip -- %f"
git config filter.lfs.clean = "git-lfs clean -- %f"

Config for good measure:

+ git config -l
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
filter.lfs.process=git-lfs filter-process --skip
filter.lfs.required=true
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge --skip -- %f
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
remote.origin.url=[https://x-token-auth:{**TOKEN**}@bitbucket.org/**USER**/**REPO**.git](https://x-token-auth:**TOKEN**@bitbucket.org/**USER**/**REPO**.git)
remote.origin.fetch=+refs/heads/master:refs/remotes/origin/master
lfs.repositoryformatversion=0

[sskjames]

I'm facing the same problem too. Hi @frankvHoof93, did you manage to solve this problem? Can you please share your findings?

[frankvHoof93]

@sskjames I finally managed to do a workaround for it.
I added a 'normal' set of credentials (username & App-password) to Jenkins.
Then, when pulling (checkout scm), I skip LFS. Afterwards I set the lfs-url to the one using the Username & App-Password, then do a LFS pull with those:

sh 'git lfs install --skip-smudge' // Required because LFS is currently failing
script {
    def scmVars = checkout scm
    // Workaround for LFS: Manually set the LFS-URL using Username & App Password, then do a pull using those instead of the OAuth
    // Remove everything up to @bitbucket.org from the Git URL (strip out the Bitbucket OAuth-credentials)
    echo "Performing LFS-Workaround to fix issue with LFS not working when using OAuth-credentials"
    def partialURL = sh(returnStdout: true, script: """echo $scmVars.GIT_URL | sed -e 's/https:\\/\\/.*:.*@bitbucket.org//'""")
    partialURL = partialURL.replace('\0', '').replace('\n', '') // Remove Null-Char & End-Line
    withCredentials([usernamePassword(credentialsId: 'Bitbucket_Basic', passwordVariable: 'BB_APP_PASS', usernameVariable: 'BB_USER')]) {
        // Generate new LFS-URL using Username & App-Password
        def LFSUrl = 'https://$BB_USER:$BB_APP_PASS@bitbucket.org' + partialURL + '/info/lfs'
        sh "git config lfs.url ${LFSUrl}" // Set as LFS-URL for the pull
        sh 'git lfs pull' // Pull LFS-files using workaround-config
    }
}

[brenc]

I ran into this too when using oAuth. A simpler workaround is to use a username and app password for the entire job then add "Git LFS pull after checkout" to the job as well. That worked for me.