OiO.lk Community platform!

Oio.lk is an excellent forum for developers, providing a wide range of resources, discussions, and support for those in the developer community. Join oio.lk today to connect with like-minded professionals, share insights, and stay updated on the latest trends and technologies in the development field.
  You need to log in or register to access the solved answers to this problem.
  • You have reached the maximum number of guest views allowed
  • Please register below to remove this limitation

How can I install python and pip via conda without internet access during environment creation in AzureML?

  • Thread starter Thread starter virginia123
  • Start date Start date
V

virginia123

Guest
I need to create an environment in AzureML, however there is no public internet access configured in the network settings of my workspace. I can access my Python packages locally on a storage account and install them. However, the conda file also requires python and pip to be installed. I know my base image already has them installed but I cannot get conda to recognize this.

Dockerfile:

Code:
FROM mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04
# Set work dir
WORKDIR /
# Define environment variables for Conda
ENV CONDA_PREFIX=/azureml-envs/myenv
ENV CONDA_DEFAULT_ENV=$CONDA_PREFIX
ENV PATH=$CONDA_PREFIX/bin:$PATH
# This is needed for mpi to locate libpython
ENV LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH

# Copy the environment.yml file to the image
COPY conda.yml .

# Set up Conda configuration
RUN conda config --add channels https://<SA>.blob.core.windows.net/<container>/<directory_to_wheel_packages> && \
    conda config --set channel_priority strict && \
    conda config --remove channels defaults && \
    conda config --set offline true
    
# Create the Conda environment
RUN conda env create -p $CONDA_PREFIX -f conda.yml && \
    rm environment.yml && \
    conda run -p $CONDA_PREFIX pip cache purge && \
    conda clean -a -y

I want to set up conda so that it doesn't check the default channels, only my storage account + the MCR base image that has python and pip installed on it.

conda file:

Code:
dependencies:
  - python=3.9
  - pip
  - pip:
    - https://<SA>.blob.core.windows.net/<container>/<directory_to_wheel_packages>/*.whl<SAS>
    - https://<SA>.blob.core.windows.net/<container>/<directory_to_wheel_packages>/*.whl<SAS>
# more packages

Error message on AML:

Code:
2024-06-27T15:06:47: Step 8/9 : RUN conda config --add channels https://<SA>.blob.core.windows.net/<container>/<directory_to_wheel_packages> &&     conda config --set channel_priority strict &&     conda config --remove channels defaults &&     conda config --set offline true
2024-06-27T15:06:47:  ---> Running in ******
2024-06-27T15:06:51: Removing intermediate container *******
2024-06-27T15:06:51:  ---> ************
2024-06-27T15:06:51: Step 9/9 : RUN conda env create -p $CONDA_PREFIX -f conda.yml &&     rm environment.yml &&     conda run -p $CONDA_PREFIX pip cache purge &&     conda clean -a -y
2024-06-27T15:06:51:  ---> Running in ***************
2024-06-27T15:06:52: Channels:
2024-06-27T15:06:52:  - 
2024-06-27T15:06:52: Platform: linux-64
2024-06-27T15:06:52: Collecting package metadata (repodata.json): ...working... done
2024-06-27T15:06:52: Solving environment: ...working... failed
2024-06-27T15:06:52: [91m
2024-06-27T15:06:52: PackagesNotFoundError: The following packages are missing from the target environment:
2024-06-27T15:06:52:   - python=3.9*
2024-06-27T15:06:52:   - pip


2024-06-27T15:06:52: The command '/bin/sh -c conda env create -p $CONDA_PREFIX -f conda.yml &&     rm conda.yml &&     conda run -p $CONDA_PREFIX pip cache purge &&     conda clean -a -y' returned a non-zero code: 1

2024-06-27T15:06:52: [0m

2024-06-27T15:06:52: CalledProcessError(1, ['docker', 'build', '-f', 'Dockerfile', '.', '-t', '***.azurecr.io/azureml/*****', '-t', '*****.azurecr.io/azureml/*********:1'])

2024-06-27T15:06:52: Building docker image failed with exit code: 1

The requirement is that these packages must not be installed locally in the workspace, because of network restrictions. Also, I must be able to do this by creating an environment and specifying a dockerfile. Could somebody point me in the right direction?
<p>I need to create an environment in AzureML, however there is no public internet access configured in the network settings of my workspace. I can access my Python packages locally on a storage account and install them. However, the conda file also requires python and pip to be installed. I know my base image already has them installed but I cannot get conda to recognize this.</p>
<p>Dockerfile:</p>
<pre><code>FROM mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04
# Set work dir
WORKDIR /
# Define environment variables for Conda
ENV CONDA_PREFIX=/azureml-envs/myenv
ENV CONDA_DEFAULT_ENV=$CONDA_PREFIX
ENV PATH=$CONDA_PREFIX/bin:$PATH
# This is needed for mpi to locate libpython
ENV LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$LD_LIBRARY_PATH

# Copy the environment.yml file to the image
COPY conda.yml .

# Set up Conda configuration
RUN conda config --add channels https://<SA>.blob.core.windows.net/<container>/<directory_to_wheel_packages> && \
conda config --set channel_priority strict && \
conda config --remove channels defaults && \
conda config --set offline true

# Create the Conda environment
RUN conda env create -p $CONDA_PREFIX -f conda.yml && \
rm environment.yml && \
conda run -p $CONDA_PREFIX pip cache purge && \
conda clean -a -y
</code></pre>
<p>I want to set up conda so that it doesn't check the default channels, only my storage account + the MCR base image that has python and pip installed on it.</p>
<p>conda file:</p>
<pre><code>dependencies:
- python=3.9
- pip
- pip:
- https://<SA>.blob.core.windows.net/<container>/<directory_to_wheel_packages>/*.whl<SAS>
- https://<SA>.blob.core.windows.net/<container>/<directory_to_wheel_packages>/*.whl<SAS>
# more packages
</code></pre>
<p>Error message on AML:</p>
<pre><code>2024-06-27T15:06:47: Step 8/9 : RUN conda config --add channels https://<SA>.blob.core.windows.net/<container>/<directory_to_wheel_packages> && conda config --set channel_priority strict && conda config --remove channels defaults && conda config --set offline true
2024-06-27T15:06:47: ---> Running in ******
2024-06-27T15:06:51: Removing intermediate container *******
2024-06-27T15:06:51: ---> ************
2024-06-27T15:06:51: Step 9/9 : RUN conda env create -p $CONDA_PREFIX -f conda.yml && rm environment.yml && conda run -p $CONDA_PREFIX pip cache purge && conda clean -a -y
2024-06-27T15:06:51: ---> Running in ***************
2024-06-27T15:06:52: Channels:
2024-06-27T15:06:52: -
2024-06-27T15:06:52: Platform: linux-64
2024-06-27T15:06:52: Collecting package metadata (repodata.json): ...working... done
2024-06-27T15:06:52: Solving environment: ...working... failed
2024-06-27T15:06:52: [91m
2024-06-27T15:06:52: PackagesNotFoundError: The following packages are missing from the target environment:
2024-06-27T15:06:52: - python=3.9*
2024-06-27T15:06:52: - pip


2024-06-27T15:06:52: The command '/bin/sh -c conda env create -p $CONDA_PREFIX -f conda.yml && rm conda.yml && conda run -p $CONDA_PREFIX pip cache purge && conda clean -a -y' returned a non-zero code: 1

2024-06-27T15:06:52: [0m

2024-06-27T15:06:52: CalledProcessError(1, ['docker', 'build', '-f', 'Dockerfile', '.', '-t', '***.azurecr.io/azureml/*****', '-t', '*****.azurecr.io/azureml/*********:1'])

2024-06-27T15:06:52: Building docker image failed with exit code: 1
</code></pre>
<p>The requirement is that these packages must not be installed locally in the workspace, because of network restrictions. Also, I must be able to do this by creating an environment and specifying a dockerfile. Could somebody point me in the right direction?</p>
 

Latest posts

Top