> ## Content Index > Fetch the complete content index at: https://anantafatur.dev/llms.txt > Use this file to discover other available public pages before exploring further. # GitLab Background Migration Stuck in Finalizing State - URL: https://anantafatur.dev/gitlab-migration-stuck-finalizing/ - Published: 2026-09-07T08:35:25.000Z - Updated: 2026-09-07T08:35:25.000Z - Description: A walkthrough of how I diagnosed and cleared a batched background migration stuck in "finalizing" that was blocking a GitLab 18.6.5 to 18.8.11 upgrade, including why the fix was safe on my instance. - Author: Ananta ## The problem I was doing a rehearsal for a GitLab upgrade. The full path I need to get through is 18.6.5 to 18.8.11 to 18.11.11 to 19.2.5 to 19.3.1, several required stops along the way based on GitLab's own upgrade path docs \[2\]. This post covers the first hop, 18.6.5 to 18.8.11, which is a direct jump with no intermediate version needed in between. I wanted to test this whole path in a rehearsal before touching anything real, and this migration issue is what I hit right at the first hop. Before any GitLab upgrade, you're supposed to check that all background migrations are done \[1\]. So I ran the usual check: ``` sudo gitlab-rake gitlab:background_migrations:status | grep -vE "finished|finalized" ``` And got this: ``` Database: main finalizing | BackfillUserDetailsFields,users,id,[] ``` One migration, `BackfillUserDetailsFields`, stuck in `finalizing`. Not `finished`, not `finalized`. GitLab won't let you upgrade past this. If you try, it throws an error like \[1\]: ``` Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active' ``` So this became a blocker before I could even continue the rehearsal. --- ## What I ruled out early First thing I checked, is GitLab actually still working? Yes. Batched background migrations run through Sidekiq in isolation, so the instance stays fully operational even with one of these stuck \[1\]. This is not an outage. It only becomes a problem when you try to upgrade. My setup uses an external PostgreSQL on TencentDB for PostgreSQL, not the bundled Omnibus database. I mention this because it changes a few things later, like how you check statement timeouts. Worth keeping in mind if you're following along with your own setup. ## Trying the documented fix first GitLab's docs give a command to manually finish a stuck migration \[1\]: ``` sudo gitlab-rake gitlab:background_migrations:finalize[BackfillUserDetailsFields,users,id,'[]'] ``` I ran it, and got this instead: ``` rake aborted! NameError: uninitialized constant Gitlab::BackgroundMigration::BackfillUserDetailsFields /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/background_migration/batched_migration.rb:235:in `job_class' ... ``` Honestly, my first thought was that something was broken in my install. But this is actually a known pattern. GitLab has a process where old migration code gets deleted from the codebase once it's considered safe to remove, sometimes called migration squashing \[3\]. The database still had a row saying "this migration is not done yet," but the actual Ruby class that does the work was already gone from the version I had installed. I confirmed this by searching the app directory directly instead of guessing: ``` find /opt/gitlab/embedded/service/gitlab-rails -iname "*queue_backfill_user_details_fields*" find /opt/gitlab/embedded/service/gitlab-rails -iname "backfill_user_details_fields.rb" ``` Both came back empty. So yeah, the code really was not there. This probably happened because I jumped across several versions during earlier upgrades, and this migration's row got left behind in an old state while the code for it kept moving forward and eventually got squashed out. ## Trying `mark_migration_complete` There's another rake task meant for cases where a regular Rails migration is missing: ``` sudo gitlab-rake gitlab:db:mark_migration_complete[20221019002459] ``` I got the version number `20221019002459` from GitLab's own merge request history for this migration \[4\], where the original migration output was pasted in the description. Not a guess, I traced it back to the actual source. Running it gave: ``` Migration version '20221019002459' is already marked complete on database main Migration version '20221019002459' is already marked complete on database ci ``` Okay, so that part was already done. But checking the background migration status again: ``` sudo gitlab-rake gitlab:background_migrations:status | grep -vE "finished|finalized" Database: main finalizing | BackfillUserDetailsFields,users,id,[] ``` Still stuck. This is the part that confused me for a bit. Turns out `mark_migration_complete` only touches the `schema_migrations` table, which tracks the regular Rails migration that *queues* the background job. It has nothing to do with `batched_background_migrations`, which is the separate table tracking the actual job's own progress. Two different tracking systems, and I only fixed one of them. --- ## The actual fix GitLab documents a manual override for exactly this situation, a batched migration that cannot run because its code is gone, and it's considered safe to skip \[1\]. You run it through the Rails console: ``` sudo gitlab-rails console ``` Then paste this in: ```ruby connection = ApplicationRecord.connection Gitlab::Database::SharedModel.using_connection(connection) do migration = Gitlab::Database::BackgroundMigration::BatchedMigration.find_for_configuration( Gitlab::Database.gitlab_schemas_for_connection(connection), 'BackfillUserDetailsFields', :users, :id, [] ) migration.batched_jobs.update_all(status: Gitlab::Database::BackgroundMigration::BatchedJob.state_machine.states['succeeded'].value) migration.update_attribute(:status, Gitlab::Database::BackgroundMigration::BatchedMigration.state_machine.states[:finished].value) end ``` This does not run the migration logic at all. It just marks all the job records as succeeded and the migration itself as finished, directly in the database. Which is exactly what I needed, since the code to actually run it was gone anyway. The GitLab docs are upfront that this can cause data loss and leave your instance in a state that's hard to recover from, and they recommend contacting support first if you have a support plan \[1\]. I did not have a support contract for this, so I went in with a backup taken first and accepted the risk, since I'd already confirmed the code was missing and this specific migration only touches a small set of profile fields, not anything core like auth or repos. After running it: ``` sudo gitlab-rake gitlab:background_migrations:status | grep -vE "finished|finalized" Database: main ``` Nothing left. Clear. ## Checking what I actually skipped This part mattered to me. Marking something "finished" without running it feels a bit like cheating, so I wanted to know what I was actually giving up. `BackfillUserDetailsFields` was supposed to copy six fields, `linkedin`, `twitter`, `skype`, `website_url`, `location`, `organization`, from the old `users` table columns into the newer `user_details` table \[4\]. This was a follow-up to an earlier migration \[5\] that set up syncing between the two. So I tried to check if there was a real gap: ```sql SELECT COUNT(*) FROM users u JOIN user_details ud ON ud.user_id = u.id WHERE (u.linkedin IS DISTINCT FROM '' AND u.linkedin IS DISTINCT FROM ud.linkedin) OR (u.twitter IS DISTINCT FROM '' AND u.twitter IS DISTINCT FROM ud.twitter) OR (u.skype IS DISTINCT FROM '' AND u.skype IS DISTINCT FROM ud.skype) OR (u.website_url IS DISTINCT FROM '' AND u.website_url IS DISTINCT FROM ud.website_url) OR (u.location IS DISTINCT FROM '' AND u.location IS DISTINCT FROM ud.location) OR (u.organization IS DISTINCT FROM '' AND u.organization IS DISTINCT FROM ud.organization); ``` Got this back: ``` pq: column u.linkedin does not exist ``` Which actually told me something useful. Those columns are already gone from the `users` table on my instance. The earlier migration \[5\] both synced the data over *and* truncated the old columns, and that had already happened, probably years before I ever hit this stuck migration. So there was nothing left in `users` for `BackfillUserDetailsFields` to copy even if it had run. The source data was already gone before I ever touched anything. That was a relief. It meant the workaround did not actually cost me anything. It was already moot by the time I got to it. --- ## References \[1\] GitLab, "Check migrations before upgrade," GitLab Docs. \[Online\]. Available: (accessed Sep. 7, 2026). \[2\] GitLab, "Plan your upgrade path," GitLab Docs. \[Online\]. Available: (accessed Sep. 7, 2026). \[3\] GitLab, "Batched background migrations," GitLab Docs. \[Online\]. Available: (accessed Sep. 7, 2026). \[4\] B. Austin, "Backfill existing user\_details fields from users table," GitLab Merge Request !101830, Oct. 22, 2022\. \[Online\]. Available: (accessed Sep. 7, 2026). \[5\] B. Austin, "Sync profile fields to user\_details and truncate," GitLab Merge Request !95107, Aug. 10, 2022\. \[Online\]. Available: (accessed Sep. 7, 2026).