GitLab Background Migration Stuck in Finalizing State

The problem

I was doing a rehearsal for a GitLab upgrade. The full path I need to get through is 18.6.5 to 18.8.11 to 18.11.11 to 19.2.5 to 19.3.1, several required stops along the way based on GitLab's own upgrade path docs [2]. This post covers the first hop, 18.6.5 to 18.8.11, which is a direct jump with no intermediate version needed in between. I wanted to test this whole path in a rehearsal before touching anything real, and this migration issue is what I hit right at the first hop.

Before any GitLab upgrade, you're supposed to check that all background migrations are done [1]. So I ran the usual check:

sudo gitlab-rake gitlab:background_migrations:status | grep -vE "finished|finalized"

And got this:

Database: main
finalizing | BackfillUserDetailsFields,users,id,[]

One migration, BackfillUserDetailsFields, stuck in finalizing. Not finished, not finalized. GitLab won't let you upgrade past this. If you try, it throws an error like [1]:

Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active'

So this became a blocker before I could even continue the rehearsal.

What I ruled out early

First thing I checked, is GitLab actually still working? Yes. Batched background migrations run through Sidekiq in isolation, so the instance stays fully operational even with one of these stuck [1]. This is not an outage. It only becomes a problem when you try to upgrade.

My setup uses an external PostgreSQL on TencentDB for PostgreSQL, not the bundled Omnibus database. I mention this because it changes a few things later, like how you check statement timeouts. Worth keeping in mind if you're following along with your own setup.

Trying the documented fix first

GitLab's docs give a command to manually finish a stuck migration [1]:

sudo gitlab-rake gitlab:background_migrations:finalize[BackfillUserDetailsFields,users,id,'[]']

I ran it, and got this instead:

rake aborted!
NameError: uninitialized constant Gitlab::BackgroundMigration::BackfillUserDetailsFields
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration.rb:235:in `job_class'
...

Honestly, my first thought was that something was broken in my install. But this is actually a known pattern. GitLab has a process where old migration code gets deleted from the codebase once it's considered safe to remove, sometimes called migration squashing [3]. The database still had a row saying "this migration is not done yet," but the actual Ruby class that does the work was already gone from the version I had installed.

I confirmed this by searching the app directory directly instead of guessing:

find /opt/gitlab/embedded/service/gitlab-rails -iname "*queue_backfill_user_details_fields*"
find /opt/gitlab/embedded/service/gitlab-rails -iname "backfill_user_details_fields.rb"

Both came back empty. So yeah, the code really was not there. This probably happened because I jumped across several versions during earlier upgrades, and this migration's row got left behind in an old state while the code for it kept moving forward and eventually got squashed out.

Trying mark_migration_complete

There's another rake task meant for cases where a regular Rails migration is missing:

sudo gitlab-rake gitlab:db:mark_migration_complete[20221019002459]

I got the version number 20221019002459 from GitLab's own merge request history for this migration [4], where the original migration output was pasted in the description. Not a guess, I traced it back to the actual source.

Running it gave:

Migration version '20221019002459' is already marked complete on database main
Migration version '20221019002459' is already marked complete on database ci

Okay, so that part was already done. But checking the background migration status again:

sudo gitlab-rake gitlab:background_migrations:status | grep -vE "finished|finalized"
Database: main
finalizing | BackfillUserDetailsFields,users,id,[]

Still stuck. This is the part that confused me for a bit. Turns out mark_migration_complete only touches the schema_migrations table, which tracks the regular Rails migration that queues the background job. It has nothing to do with batched_background_migrations, which is the separate table tracking the actual job's own progress. Two different tracking systems, and I only fixed one of them.

The actual fix

GitLab documents a manual override for exactly this situation, a batched migration that cannot run because its code is gone, and it's considered safe to skip [1]. You run it through the Rails console:

sudo gitlab-rails console

Then paste this in:

connection = ApplicationRecord.connection

Gitlab::Database::SharedModel.using_connection(connection) do
  migration = Gitlab::Database::BackgroundMigration::BatchedMigration.find_for_configuration(
    Gitlab::Database.gitlab_schemas_for_connection(connection),
    'BackfillUserDetailsFields',
    :users,
    :id,
    []
  )

  migration.batched_jobs.update_all(status: Gitlab::Database::BackgroundMigration::BatchedJob.state_machine.states['succeeded'].value)
  migration.update_attribute(:status, Gitlab::Database::BackgroundMigration::BatchedMigration.state_machine.states[:finished].value)
end

This does not run the migration logic at all. It just marks all the job records as succeeded and the migration itself as finished, directly in the database. Which is exactly what I needed, since the code to actually run it was gone anyway.

The GitLab docs are upfront that this can cause data loss and leave your instance in a state that's hard to recover from, and they recommend contacting support first if you have a support plan [1]. I did not have a support contract for this, so I went in with a backup taken first and accepted the risk, since I'd already confirmed the code was missing and this specific migration only touches a small set of profile fields, not anything core like auth or repos.

After running it:

sudo gitlab-rake gitlab:background_migrations:status | grep -vE "finished|finalized"
Database: main

Nothing left. Clear.

Checking what I actually skipped

This part mattered to me. Marking something "finished" without running it feels a bit like cheating, so I wanted to know what I was actually giving up.

BackfillUserDetailsFields was supposed to copy six fields, linkedin, twitter, skype, website_url, location, organization, from the old users table columns into the newer user_details table [4]. This was a follow-up to an earlier migration [5] that set up syncing between the two.

So I tried to check if there was a real gap:

SELECT COUNT(*) FROM users u
JOIN user_details ud ON ud.user_id = u.id
WHERE (u.linkedin IS DISTINCT FROM '' AND u.linkedin IS DISTINCT FROM ud.linkedin)
   OR (u.twitter IS DISTINCT FROM '' AND u.twitter IS DISTINCT FROM ud.twitter)
   OR (u.skype IS DISTINCT FROM '' AND u.skype IS DISTINCT FROM ud.skype)
   OR (u.website_url IS DISTINCT FROM '' AND u.website_url IS DISTINCT FROM ud.website_url)
   OR (u.location IS DISTINCT FROM '' AND u.location IS DISTINCT FROM ud.location)
   OR (u.organization IS DISTINCT FROM '' AND u.organization IS DISTINCT FROM ud.organization);

Got this back:

pq: column u.linkedin does not exist

Which actually told me something useful. Those columns are already gone from the users table on my instance. The earlier migration [5] both synced the data over and truncated the old columns, and that had already happened, probably years before I ever hit this stuck migration. So there was nothing left in users for BackfillUserDetailsFields to copy even if it had run. The source data was already gone before I ever touched anything.

That was a relief. It meant the workaround did not actually cost me anything. It was already moot by the time I got to it.

References

[1] GitLab, "Check migrations before upgrade," GitLab Docs. [Online]. Available: https://docs.gitlab.com/ee/update/background_migrations.html (accessed Sep. 7, 2026).

[2] GitLab, "Plan your upgrade path," GitLab Docs. [Online]. Available: https://docs.gitlab.com/ee/update/upgrade_paths.html (accessed Sep. 7, 2026).

[3] GitLab, "Batched background migrations," GitLab Docs. [Online]. Available: https://docs.gitlab.com/ee/development/database/batched_background_migrations.html (accessed Sep. 7, 2026).

[4] B. Austin, "Backfill existing user_details fields from users table," GitLab Merge Request !101830, Oct. 22, 2022. [Online]. Available: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/101830 (accessed Sep. 7, 2026).

[5] B. Austin, "Sync profile fields to user_details and truncate," GitLab Merge Request !95107, Aug. 10, 2022. [Online]. Available: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/95107 (accessed Sep. 7, 2026).