Skip to content

Commit fe79ceb

Browse files
senekorsenekor
authored
Rewrite Dockerfile to share layers with analyzer (#195)
The main goal of this is to bring the Dockerfiles of the test-runner and analyzer in sync, so they can share the big layers containing the Rust toolchain and local cargo registry. Some incidental changes / improvements that were made: * Use an official base image with a nightly Rust toolchain, but pinned to a specific hash. This avoids our cache being invalidated every day, without us needing to copy-paste the upstream build script. * Make local-registry/Cargo.toml more readable for users by moving irrelevant stuff to the bottom. Exercises contain a link to this file, so that students can check themselves which crates are available. * Use heredoc in the Dockerfile for better readability and to reduce the number of intermediate layers generated during a build.
1 parent 86f60ef commit fe79ceb

3 files changed

Lines changed: 71 additions & 142 deletions

File tree

‎Dockerfile‎

Lines changed: 58 additions & 131 deletions
Original file line numberDiff line numberDiff line change
@@ -1,147 +1,74 @@
1-
# always build this using the latest stable release
2-
FROM rust:1.95.0 AS build-base
1+
############################# START SHARED LAYERS #############################
32

4-
ARG JQ_VERSION=1.6
5-
ARG JQ_URL=https://github.com/stedolan/jq/releases/download/jq-${JQ_VERSION}/jq-linux64
6-
# download jq
7-
RUN curl -L -o /usr/local/bin/jq "${JQ_URL}" \
8-
&& chmod +x /usr/local/bin/jq
3+
# IMPORTANT: This part of the build is shared between the test-runner and
4+
# analyzer. It takes a relatively large amount of space: 850 MB for the Rust
5+
# toolchain and 40 MB for the local cargo registry. Therefore, it's important
6+
# to keep this in sync between the two images. A slight mismatch in these layers
7+
# would lead to douple the storage requirement on Exercism's servers.
98

10-
# install cargo-local-registry dependencies
11-
RUN apt-get update && apt-get install -y gcc openssl cmake
9+
FROM rust:1.95.0 AS build-local-registry
1210

11+
WORKDIR /work
12+
COPY local-registry/Cargo.toml .
1313

14-
FROM build-base AS build-rust-test-runner
14+
RUN <<EOF
15+
set -eux
16+
apt-get update
17+
apt-get install -y gcc openssl cmake
18+
cargo install --locked cargo-local-registry
19+
cargo generate-lockfile
20+
cargo local-registry sync Cargo.lock /local-registry
21+
EOF
1522

16-
RUN mkdir -p /rust-test-runner/src
17-
ENV wd=/rust-test-runner
18-
WORKDIR ${wd}
19-
COPY Cargo.* ./
20-
# for caching, we want to download and build all the dependencies before copying
21-
# any of the real source files. We therefore build an empty dummy library,
22-
# then remove it.
23-
RUN echo '// dummy file' > src/lib.rs
24-
RUN cargo build
25-
# now get rid of the stub and copy the real source files
26-
RUN rm src/lib.rs
27-
COPY src/* src/
28-
# build the executable
29-
RUN cargo build --release
30-
31-
32-
FROM build-base AS build-cargo-local-registry
33-
34-
# install cargo-local-registry
35-
RUN cargo install --locked cargo-local-registry
36-
# download popular crates to local registry
37-
WORKDIR /local-registry
38-
COPY local-registry/* ./
39-
RUN cargo generate-lockfile && cargo local-registry --sync Cargo.lock .
40-
41-
42-
# As of Dec 2019, we need to use the nightly toolchain to get JSON test output
23+
# As of April 2026, we need to use the nightly toolchain to get JSON test output
4324
# tracking issue: https://github.com/rust-lang/rust/issues/49359
25+
#
26+
# The official docker image for the nightly Rust toolchain is updated every
27+
# day. We don't want to invalidate the build cache that often, so we pin it
28+
# to a specific hash. To update, go to the following page, then navigate to
29+
# "nightly-slim", select "linux/amd64" and copy the manifest digest.
30+
# https://hub.docker.com/r/rustlang/rust/tags
31+
#
32+
FROM docker.io/rustlang/rust@sha256:3444fefbb69afbff45c0722c8045404c8e7f369c5202e916bd94f665b69f1b1c AS rust-nightly-with-local-registry
4433

45-
# Official docker images with pinned nightly versions are not provided, but we
46-
# want to pin the nightly version to avoid unnecessary cache misses. To achieve
47-
# this, we copy the source of the official rust docker images and replace the
48-
# version tag with a nightly one, pinned to a specific date.
34+
# add local registry
35+
COPY --from=build-local-registry /local-registry /local-registry
36+
RUN <<EOF
37+
echo "\
38+
[source.crates-io]
39+
registry = 'sparse+https://index.crates.io/'
40+
replace-with = 'local-registry'
4941
50-
# official Dockerfile source:
51-
# https://github.com/rust-lang/docker-rust/blob/master/stable/trixie/slim/Dockerfile
42+
[source.local-registry]
43+
local-registry = '/local-registry'" >> $CARGO_HOME/config.toml
44+
EOF
5245

53-
################ start-copy-pasta ################
46+
############################## END SHARED LAYERS ##############################
5447

55-
FROM debian:trixie-slim
5648

57-
LABEL org.opencontainers.image.source=https://github.com/rust-lang/docker-rust
49+
FROM rust:1.95.0 AS build
50+
51+
WORKDIR /work
52+
COPY Cargo.* ./
53+
# for caching, we want to download and build all the dependencies before copying
54+
# any of the real source files. We therefore build an empty dummy library,
55+
# then remove it.
56+
RUN <<EOF
57+
mkdir --parents src
58+
touch src/lib.rs
59+
cargo build --release
60+
rm src/lib.rs
61+
EOF
62+
# now build the real test-runner
63+
COPY src/* src/
64+
RUN touch src/lib.rs && cargo build --release
5865

59-
ENV RUSTUP_HOME=/usr/local/rustup \
60-
CARGO_HOME=/usr/local/cargo \
61-
PATH=/usr/local/cargo/bin:$PATH \
62-
RUST_VERSION=nightly-2026-04-17
63-
# ~~~~~~~~^~~~~~~~~~
64-
# pin version here
6566

66-
RUN set -eux; \
67-
\
68-
apt-get update; \
69-
apt-get install -y --no-install-recommends \
70-
ca-certificates \
71-
gcc \
72-
libc6-dev \
73-
wget \
74-
; \
75-
\
76-
arch="$(dpkg --print-architecture)"; \
77-
case "$arch" in \
78-
'amd64') \
79-
rustArch='x86_64-unknown-linux-gnu'; \
80-
rustupSha256='20a06e644b0d9bd2fbdbfd52d42540bdde820ea7df86e92e533c073da0cdd43c'; \
81-
;; \
82-
'armhf') \
83-
rustArch='armv7-unknown-linux-gnueabihf'; \
84-
rustupSha256='3b8daab6cc3135f2cd4b12919559e6adaee73a2fbefb830fadf0405c20231d61'; \
85-
;; \
86-
'arm64') \
87-
rustArch='aarch64-unknown-linux-gnu'; \
88-
rustupSha256='e3853c5a252fca15252d07cb23a1bdd9377a8c6f3efa01531109281ae47f841c'; \
89-
;; \
90-
'i386') \
91-
rustArch='i686-unknown-linux-gnu'; \
92-
rustupSha256='a5db2c4b29d23e9b318b955dd0337d6b52e93933608469085c924e0d05b1df1f'; \
93-
;; \
94-
'ppc64el') \
95-
rustArch='powerpc64le-unknown-linux-gnu'; \
96-
rustupSha256='acd89c42b47c93bd4266163a7b05d3f26287d5148413c0d47b2e8a7aa67c9dc0'; \
97-
;; \
98-
's390x') \
99-
rustArch='s390x-unknown-linux-gnu'; \
100-
rustupSha256='726b7fd5d8805e73eab4a024a2889f8859d5a44e36041abac0a2436a52d42572'; \
101-
;; \
102-
'riscv64') \
103-
rustArch='riscv64gc-unknown-linux-gnu'; \
104-
rustupSha256='09e64cc1b7a3e99adaa15dd2d46a3aad9d44d71041e2a96100d165c98a8fd7a7'; \
105-
;; \
106-
*) \
107-
echo >&2 "unsupported architecture: $arch"; \
108-
exit 1; \
109-
;; \
110-
esac; \
111-
\
112-
url="https://static.rust-lang.org/rustup/archive/1.28.2/${rustArch}/rustup-init"; \
113-
wget --progress=dot:giga "$url"; \
114-
echo "${rustupSha256} *rustup-init" | sha256sum -c -; \
115-
\
116-
chmod +x rustup-init; \
117-
./rustup-init -y --no-modify-path --profile minimal --default-toolchain $RUST_VERSION --default-host ${rustArch}; \
118-
rm rustup-init; \
119-
chmod -R a+w $RUSTUP_HOME $CARGO_HOME; \
120-
\
121-
apt-get remove -y --auto-remove \
122-
wget \
123-
; \
124-
rm -rf /var/lib/apt/lists/*; \
125-
\
126-
rustup --version; \
127-
cargo --version; \
128-
rustc --version;
67+
FROM rust-nightly-with-local-registry
12968

130-
################ end-copy-pasta ################
69+
RUN apt-get update && apt-get install -y jq && rm -rf /var/lib/apt/lists/*
13170

132-
ENV wd=/opt/test-runner
133-
RUN mkdir -p ${wd}/bin
134-
WORKDIR ${wd}
135-
COPY --from=build-rust-test-runner /rust-test-runner/target/release/rust_test_runner bin
136-
COPY --from=build-base /usr/local/bin/jq /usr/local/bin
137-
COPY --from=build-cargo-local-registry /local-registry local-registry/
138-
# configure local-registry
139-
RUN echo '[source.crates-io]\n\
140-
registry = "https://github.com/rust-lang/crates.io-index"\n\
141-
replace-with = "local-registry"\n\
142-
\n\
143-
[source.local-registry]\n\
144-
local-registry = "/opt/test-runner/local-registry/"\n' >> $CARGO_HOME/config.toml
145-
# set entrypoint
146-
COPY bin/run.sh bin
71+
WORKDIR /opt/test-runner
72+
COPY --from=build /work/target/release/rust_test_runner bin/
73+
COPY bin/run.sh bin/
14774
ENTRYPOINT ["bin/run.sh"]

‎local-registry/Cargo.toml‎

Lines changed: 13 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,11 @@
1-
[package]
2-
name = "dummy_package"
3-
version = "0.1.0"
4-
edition = "2021"
5-
6-
[lib]
7-
path = "dummy.rs"
8-
91
# IMPORTANT: These dependencies should be kept in sync with the ones at
102
# https://github.com/exercism/rust-analyzer/blob/main/local-registry/Cargo.toml
113
#
124
# Some crates are used by a large number of solutions. For example, when a crate
135
# is required or already included in the exercise skeleton. For those crates,
14-
# we should *never* drop support for a given major version. We can continue to
15-
# support old and new versions by giving the old ones an alias. The downside is
16-
# an increase in the size of the local registry.
6+
# we should NEVER drop support for a given major version. They have been marked
7+
# with a "# pin" comment. We can continue to support old and new versions by
8+
# giving the old ones an alias. An example is the rand_09 crate below.
179
#
1810
[dependencies]
1911
anyhow = "1.0.102" # pin
@@ -119,3 +111,13 @@ unzip-n = "0.1.4"
119111
uuid = "1.23.1"
120112
voca_rs = "1.15.2"
121113
xvii = "0.4.0"
114+
115+
# The following is only needed so the Cargo.toml defines a valid package.
116+
117+
[package]
118+
name = "dummy_package"
119+
version = "0.1.0"
120+
edition = "2024"
121+
122+
[lib]
123+
path = "/dev/null"

‎local-registry/dummy.rs‎

Whitespace-only changes.

0 commit comments

Comments
 (0)