Skip to content

[ML] Update Linux builder Docker container for 9.0 #2773

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 42 commits into from
Nov 11, 2024
Merged

[ML] Update Linux builder Docker container for 9.0 #2773

merged 42 commits into from
Nov 11, 2024

Conversation

valeriy42
Copy link
Contributor

@valeriy42 valeriy42 commented Oct 28, 2024 •
edited
Loading

This PR updates all Linux-related Dockerfiles and tutorials with the latest dependencies, libraries, and C++ compiler versions.

Changes

  • Dockerfile Updates:

    • The base image is changed from centos:7 to rockylinux:8 to maintain compatibility with RHEL in the future.
    • Updated all GCC versions to use GCC 13.3 and binutils 2.41 for enhanced C++17 support and security hardening.
    • Other updated libraries:
      • binutils 2.37 -> 2.41
      • libxml2 2.9.14 -> 2.10.4.
      • patchelf 0.13 -> 0.18.0
      • python 3.10.0 -> 3.12.7
      • cmake 3.19.2 -> 3.30.5

  • Tutorial Updates:

    • Revised instructions for setting up the Linux build environment to match new dependencies and configurations.
    • Provided updated example commands for package installation compatible with Rocky Linux 8 and similar distributions.

  • Buildkite:

    • Updated the CI scripts to use the newest version of the Docker containers.

@valeriy42 valeriy42 closed this Oct 29, 2024
@valeriy42 valeriy42 reopened this Oct 30, 2024
@valeriy42 valeriy42 self-assigned this Oct 30, 2024
@valeriy42 valeriy42 requested a review from edsavage October 30, 2024 08:39
@valeriy42 valeriy42 marked this pull request as ready for review October 30, 2024 09:05
edsavage
edsavage reviewed Oct 30, 2024
View reviewed changes
edsavage
edsavage reviewed Oct 30, 2024
View reviewed changes
edsavage
edsavage reviewed Oct 30, 2024
View reviewed changes
edsavage
edsavage reviewed Oct 30, 2024
View reviewed changes
edsavage
edsavage approved these changes Oct 30, 2024
View reviewed changes
Copy link
Contributor

@edsavage edsavage left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

I've just left a few comments here and there.

@valeriy42
review comments
a509595
@valeriy42
update aarch64 cross dependencz on S3
85850eb
@valeriy42
add compiler flags to fix numerical issues on aarch64
b210471
@valeriy42
use newest aarch64 buildkite image
e27d01c
@valeriy42
Merge branch 'linux-docker-upgrade' of https://github.com/valeriy42/m…
6a8d8d2 
…l-cpp into linux-docker-upgrade
@valeriy42
update aarch64 image prefix from Amazon Linux to AlmaLinux 8
cce2b2d
@valeriy42
test image name for aarch64
49a7811
@valeriy42
temporary remove dependencies and build steps
4febf61
@valeriy42
fix pipeline
ece33b8
@valeriy42
fixing pipeline
74e1207
@valeriy42
update aarch64 image prefix to remove unnecessary prefix
3d89ec1
@valeriy42
update aarch64 image prefix to use 'ci-' prefix
f88c877
@valeriy42
update aarch64 image prefix to use 'core-' prefix
aa876c3
@valeriy42
refactor pipeline steps and update image prefix for x86_64 tests
69e6748
@valeriy42
update aarch64 image prefix to use 'core-' prefix
ff52dbd
@valeriy42
restore dependency on check_style in build pipeline
8b92ca9
@valeriy42
update x86_64 image prefix to use 'ci-amazon-2023'
011b38a
@valeriy42
update provider to GCP and change image prefix to core-almalinux-8 fo…
da4d9b1 
…r x86_64 tests
@valeriy42
update provider to AWS in x86_64 test pipeline
b9fd75a
@elastic-sonarqube Elastic SonarQube
Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

@valeriy42 valeriy42 merged commit b9a79b2 into elastic:main Nov 11, 2024
16 checks passed
jan-elastic added a commit that referenced this pull request Nov 12, 2024
@jan-elastic
Revert "[ML] Update Linux builder Docker container for 9.0 (#2773)"
77bcd06 
This reverts commit b9a79b2.
jan-elastic added a commit that referenced this pull request Nov 12, 2024
@jan-elastic
Revert "[ML] Update Linux builder Docker container for 9.0 (#2773)" (#…
56cac81 
…2776)

This reverts commit b9a79b2.
@valeriy42 valeriy42 mentioned this pull request Nov 28, 2024
Merged
valeriy42 added a commit that referenced this pull request Dec 3, 2024
@valeriy42
[ML] Update Linux builder Docker container for 9.0 (#2785)
479203a 
This is a rework of #2773 . Additionally, I removed the dependency on liblzma when building the boost library since some of the production docker containers don't have liblzma installed.

I added arguments like --without-lzma, but for Boost, it didn't lead to the desired effect. Hence, I had to resolve to temporarily rename the system libraries:

RUN \
  for i in $(ls /lib64/libzstd.*); do mv $i "$i.bak" ; done && \
  for i in $(ls /lib64/liblzma.*); do mv $i "$i.bak" ; done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@edsavage edsavage edsavage approved these changes

Assignees

@valeriy42 valeriy42

Labels
>build :ml v9.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@valeriy42 @edsavage