Skip to content

Job logs

  • Tier: Free, Premium, Ultimate
  • Offering: GitLab Self-Managed

Job logs are sent by a runner while it's processing a job. You can see logs in places like job pages, pipelines, and email notifications.

Data flow

In general, there are two states for job logs: log and archived log. In the following table you can see the phases a log goes through:

Phase State Condition Data flow Stored path
1: patching log When a job is running Runner => Puma => file storage #{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log
2: archiving archived log After a job is finished Sidekiq moves log to artifacts folder #{ROOT_PATH}/gitlab-rails/shared/artifacts/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log
3: uploading archived log After a log is archived Sidekiq moves archived log to object storage (if configured) #{bucket_name}/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log

The ROOT_PATH varies per environment:

  • For the Linux package it's /var/opt/gitlab.
  • For self-compiled installations it's /home/git/gitlab.

Changing the job logs local location

For Docker installations, you can change the path where your data is mounted. For the Helm chart, use object storage.

To change the location where the job logs are stored:

Linux package (Omnibus)

  1. Optional. If you have existing job logs, pause continuous integration data processing by temporarily stopping Sidekiq:

    sudo gitlab-ctl stop sidekiq

  2. Set the new storage location in /etc/gitlab/gitlab.rb:

    gitlab_ci['builds_directory'] = '/mnt/gitlab-ci/builds'

  3. Save the file and reconfigure GitLab:

    sudo gitlab-ctl reconfigure

  4. Use rsync to move job logs from the current location to the new location:

    sudo rsync -avzh --remove-source-files --ignore-existing --progress /var/opt/gitlab/gitlab-ci/builds/ /mnt/gitlab-ci/builds/

    Use --ignore-existing so you don't override new job logs with older versions of the same log.

  5. If you opted to pause the continuous integration data processing, you can start Sidekiq again:

    sudo gitlab-ctl start sidekiq

  6. Remove the old job logs storage location:

    sudo rm -rf /var/opt/gitlab/gitlab-ci/builds

Self-compiled (source)

  1. Optional. If you have existing job logs, pause continuous integration data processing by temporarily stopping Sidekiq:

    # For systems running systemd
sudo systemctl stop gitlab-sidekiq

# For systems running SysV init
sudo service gitlab stop

  2. Edit /home/git/gitlab/config/gitlab.yml to set the new storage location:

    production: &base
  gitlab_ci:
    builds_path: /mnt/gitlab-ci/builds

  3. Save the file and restart GitLab:

    # For systems running systemd
sudo systemctl restart gitlab.target

# For systems running SysV init
sudo service gitlab restart

  4. Use rsync to move job logs from the current location to the new location:

    sudo rsync -avzh --remove-source-files --ignore-existing --progress /home/git/gitlab/builds/ /mnt/gitlab-ci/builds/

    Use --ignore-existing so you don't override new job logs with older versions of the same log.

  5. If you opted to pause the continuous integration data processing, you can start Sidekiq again:

    # For systems running systemd
sudo systemctl start gitlab-sidekiq

# For systems running SysV init
sudo service gitlab start

  6. Remove the old job logs storage location:

    sudo rm -rf /home/git/gitlab/builds

Uploading logs to object storage

Archived logs are considered as job artifacts. Therefore, when you set up the object storage integration, job logs are automatically migrated to it along with the other job artifacts.

See "Phase 3: uploading" in Data flow to learn about the process.

Maximum log file size

The job log file size limit in GitLab is 100 megabytes by default. Any job that exceeds the limit is marked as failed, and dropped by the runner. For more details, see Maximum file size for job logs.

Prevent local disk usage

If you want to avoid any local disk usage for job logs, you can do so using one of the following options:

How to remove job logs

There isn't a way to automatically expire old job logs. However, it's safe to remove them if they're taking up too much space. If you remove the logs manually, the job output in the UI is empty.

For details on how to delete job logs by using GitLab CLI, see Delete job logs.

Alternatively, you can delete job logs with shell commands. For example, to delete all job logs older than 60 days, run the following command from a shell in your GitLab instance.

For the Helm chart, use the storage management tools provided with your object storage.

The following command permanently deletes the log files and is irreversible.

Linux package (Omnibus)

find /var/opt/gitlab/gitlab-rails/shared/artifacts -name "job.log" -mtime +60 -delete

Docker

Assuming you mounted /var/opt/gitlab to /srv/gitlab:

find /srv/gitlab/gitlab-rails/shared/artifacts -name "job.log" -mtime +60 -delete

Self-compiled (source)

find /home/git/gitlab/shared/artifacts -name "job.log" -mtime +60 -delete

After the logs are deleted, you can find any broken file references by running the Rake task that checks the integrity of the uploaded files. For more information, see how to delete references to missing artifacts.

Incremental logging

Incremental logging changes how job logs are processed and stored, improving performance in scaled-out deployments.

By default, job logs are sent from GitLab Runner in chunks and cached temporarily on disk. After the job completes, a background job archives the log to the artifacts directory or to object storage if configured.

With incremental logging, logs are stored in Redis and a persistent store instead of file storage. This approach:

  • Prevents local disk usage for job logs.
  • Eliminates the need for NFS sharing between Rails and Sidekiq servers.
  • Improves performance in multi-node installations.

The incremental logging process uses Redis as temporary storage and follows this flow:

  1. The runner picks a job from GitLab.
  2. The runner sends a piece of log to GitLab.
  3. GitLab appends the data to Redis in the Gitlab::Redis::TraceChunks namespace.
  4. After the data in Redis reaches 128 KB, the data is flushed to a persistent store.
  5. The above steps repeat until the job is finished.
  6. After the job is finished, GitLab schedules a Sidekiq worker to archive the log.
  7. The Sidekiq worker archives the log to object storage and cleans up temporary data.

Redis Cluster is not supported with incremental logging. For more information, see issue 224171.

Configure incremental logging

Before you turn on incremental logging, you must configure object storage for CI/CD artifacts, logs, and builds. After incremental logging is turned on, files cannot be written to disk, and there is no protection against misconfiguration.

When you turn on incremental logging, running jobs' logs continue to be written to disk, but new jobs use incremental logging.

When you turn off incremental logging, running jobs continue to use incremental logging, but new jobs write to the disk.

To configure incremental logging: