OiO.lk Community platform!

Oio.lk is an excellent forum for developers, providing a wide range of resources, discussions, and support for those in the developer community. Join oio.lk today to connect with like-minded professionals, share insights, and stay updated on the latest trends and technologies in the development field.
  You need to log in or register to access the solved answers to this problem.
  • You have reached the maximum number of guest views allowed
  • Please register below to remove this limitation

PyTorch: `set_rng_state` vs `manual_seed`

  • Thread starter Thread starter Shay
  • Start date Start date
S

Shay

Guest
In the context of PyTorch: I've been wanting to set the seed temporarily and then revert the random state to what it originally was.

I've thus been looking at various sections of the PyTorch docs. In the section for set_rng_state:

This function only works for CPU. For CUDA, please use torch.manual_seed(), which works for both CPU and CUDA.

I do want it to apply to GPU as well so I went for manual_seed (as in: I set it to something else), mentioned both in the Reproducibility section and its own, but before doing so I attempted to save the current seed as returned by torch.seed, and then eventually restore it. Something like this:

Code:
import torch

orig_seed = torch.seed()
torch.manual_seed(seed)
torch.manual_seed(orig_seed)

I tried performing some RNG-based operation with and without running this to ensure I get the same results, but I didn't.

This is the code I ran:

Code:
import torch
import contextlib

@contextlib.contextmanager
def temp_seed(seed: int):
    orig_seed = torch.seed()
    torch.manual_seed(seed)
    yield
    torch.manual_seed(orig_seed)

torch.manual_seed(1010)

torch.randint(1, 10, (5,))

with temp_seed(42):
  torch.randint(1, 10, (5,))

torch.randint(1, 10, (5,))

And then, on a new interpreter instance, I ran this code:

Code:
import torch
import contextlib

@contextlib.contextmanager
def temp_seed(seed: int):
    orig_seed = torch.seed()
    torch.manual_seed(seed)
    yield
    torch.manual_seed(orig_seed)


torch.manual_seed(1010)

torch.randint(1, 10, (5,))
torch.randint(1, 10, (5,))

The latter block resulted in the following two tensors:

Code:
tensor([4, 8, 5, 2, 1])
tensor([1, 8, 9, 6, 2])

The former, in these three:

Code:
tensor([4, 8, 5, 2, 1])
tensor([7, 6, 8, 5, 1])
tensor([6, 7, 1, 5, 6])

Specifically, tensor([1, 8, 9, 6, 2]) != tensor([6, 7, 1, 5, 6]) so it didn't work.
Admittedly, I saw no mention of manual_seed(seed) resulting in a noop (and indeed, running torch.manual_seed(torch.seed()) appears to cause changes in the RNG state as returned by torch.get_rng_state), so clearly I misunderstood the way this works.

To further experiment and potentially gain some more insight, after finding the aforementioned get/set_rng_state in the docs, I tried this code instead:

Code:
import torch
import contextlib

@contextlib.contextmanager
def temp_seed(seed: int):
    orig_state = torch.get_rng_state()
    torch.manual_seed(seed)
    yield
    torch.set_rng_state(orig_state)


torch.manual_seed(1010)

torch.randint(1, 10, (5,))

with temp_seed(42):
  torch.randint(1, 10, (5,))

torch.randint(1, 10, (5,))

Followed again by this:

Code:
import torch
import contextlib

@contextlib.contextmanager
def temp_seed(seed: int):
    orig_state = torch.get_rng_state()
    torch.manual_seed(seed)
    yield
    torch.set_rng_state(orig_state)


torch.manual_seed(1010)

torch.randint(1, 10, (5,))
torch.randint(1, 10, (5,))

Except this time the results were, as I'd hoped, these respectively:

Code:
tensor([4, 8, 5, 2, 1])
tensor([7, 6, 8, 5, 1])
tensor([1, 8, 9, 6, 2])

Code:
tensor([4, 8, 5, 2, 1])
tensor([1, 8, 9, 6, 2])

So indeed, the temporary seed-change had occurred only temporarily, and the random state was then apparently restored.
However, if I understand correctly, as described above, this'd only work on the CPU. I was wondering what might be the method to accomplish this in a manner that'd affect the GPU as well.
I also realize that different components might use different seeds, so numpy has its numpy.random.seed and Python has random.seed, IIRC, and I recognize that nondeterminism might occur even if I do set all three of those; but, even just to understand wassup a little better, I'd like to make it work in this minimal & limited example.

To summarize:
In PyTorch, what might be a way to save-and-restore the RNG state, but in a manner that'd affect both the CPU and the GPU, and is preferably as pythonic (and torchonic 😛) as possible, or in other words, the "intended method"?

I'll also note that I saw mention of torch.cuda.get/set_rng_state, and maybe I could just do everything I do with the "CPU RNG state" with the "Cuda RNG state" as well; I just wonder if that's the intended use, and didn't see a very direct mention of this in the docs.
<p>In the context of PyTorch: I've been wanting to set the seed temporarily and then revert the random state to what it originally was.</p>
<p>I've thus been looking at various sections of the PyTorch docs. In the section for <a href="https://pytorch.org/docs/stable/generated/torch.set_rng_state.html#torch.set_rng_state" rel="nofollow noreferrer"><code>set_rng_state</code></a>:</p>
<blockquote>
<p>This function only works for CPU. For CUDA, please use torch.manual_seed(), which works for both CPU and CUDA.</p>
</blockquote>
<p>I do want it to apply to GPU as well so I went for <code>manual_seed</code> (as in: I set it to something else), mentioned both in the <a href="https://pytorch.org/docs/stable/notes/randomness.html#reproducibility" rel="nofollow noreferrer">Reproducibility</a> section and <a href="https://pytorch.org/docs/stable/generated/torch.manual_seed.html#torch.manual_seed" rel="nofollow noreferrer">its own</a>, but before doing so I attempted to save the current seed as returned by <code>torch.seed</code>, and then eventually restore it. Something like this:</p>
<pre><code>import torch

orig_seed = torch.seed()
torch.manual_seed(seed)
torch.manual_seed(orig_seed)
</code></pre>
<p>I tried performing some RNG-based operation with and without running this to ensure I get the same results, but I didn't.</p>
<p>This is the code I ran:</p>
<pre><code>import torch
import contextlib

@contextlib.contextmanager
def temp_seed(seed: int):
orig_seed = torch.seed()
torch.manual_seed(seed)
yield
torch.manual_seed(orig_seed)

torch.manual_seed(1010)

torch.randint(1, 10, (5,))

with temp_seed(42):
torch.randint(1, 10, (5,))

torch.randint(1, 10, (5,))
</code></pre>
<p>And then, on a new interpreter instance, I ran this code:</p>
<pre><code>import torch
import contextlib

@contextlib.contextmanager
def temp_seed(seed: int):
orig_seed = torch.seed()
torch.manual_seed(seed)
yield
torch.manual_seed(orig_seed)


torch.manual_seed(1010)

torch.randint(1, 10, (5,))
torch.randint(1, 10, (5,))
</code></pre>
<p>The latter block resulted in the following two tensors:</p>
<pre><code>tensor([4, 8, 5, 2, 1])
tensor([1, 8, 9, 6, 2])
</code></pre>
<p>The former, in these three:</p>
<pre><code>tensor([4, 8, 5, 2, 1])
tensor([7, 6, 8, 5, 1])
tensor([6, 7, 1, 5, 6])
</code></pre>
<p>Specifically, <code>tensor([1, 8, 9, 6, 2]) != tensor([6, 7, 1, 5, 6])</code> so it didn't work.<br />
Admittedly, I saw no mention of <code>manual_seed(seed)</code> resulting in a noop (and indeed, running <code>torch.manual_seed(torch.seed())</code> appears to cause changes in the RNG state as returned by <code>torch.get_rng_state</code>), so clearly I misunderstood the way this works.</p>
<p>To further experiment and potentially gain some more insight, after finding the aforementioned <a href="https://pytorch.org/docs/stable/torch.html#torch.get_rng_state" rel="nofollow noreferrer"><code>get</code></a>/<a href="https://pytorch.org/docs/stable/torch.html#torch.set_rng_state" rel="nofollow noreferrer"><code>set</code></a><code>_rng_state</code> in the docs, I tried this code instead:</p>
<pre><code>import torch
import contextlib

@contextlib.contextmanager
def temp_seed(seed: int):
orig_state = torch.get_rng_state()
torch.manual_seed(seed)
yield
torch.set_rng_state(orig_state)


torch.manual_seed(1010)

torch.randint(1, 10, (5,))

with temp_seed(42):
torch.randint(1, 10, (5,))

torch.randint(1, 10, (5,))
</code></pre>
<p>Followed again by this:</p>
<pre><code>import torch
import contextlib

@contextlib.contextmanager
def temp_seed(seed: int):
orig_state = torch.get_rng_state()
torch.manual_seed(seed)
yield
torch.set_rng_state(orig_state)


torch.manual_seed(1010)

torch.randint(1, 10, (5,))
torch.randint(1, 10, (5,))
</code></pre>
<p>Except this time the results were, as I'd hoped, these respectively:</p>
<pre><code>tensor([4, 8, 5, 2, 1])
tensor([7, 6, 8, 5, 1])
tensor([1, 8, 9, 6, 2])
</code></pre>
<pre><code>tensor([4, 8, 5, 2, 1])
tensor([1, 8, 9, 6, 2])
</code></pre>
<p>So indeed, the temporary seed-change had occurred only temporarily, and the random state was then apparently restored.<br />
However, if I understand correctly, as described above, this'd only work on the CPU. I was wondering what might be the method to accomplish this in a manner that'd affect the GPU as well.<br />
I also realize that different components might use different seeds, so numpy has its <code>numpy.random.seed</code> and Python has <code>random.seed</code>, IIRC, and I recognize that nondeterminism might occur even if I do set all three of those; but, even just to understand wassup a little better, I'd like to make it work in this minimal & limited example.</p>
<p>To summarize:<br />
In PyTorch, what might be a way to save-and-restore the RNG state, but in a manner that'd affect both the CPU and the GPU, and is preferably as pythonic (and torchonic 😛) as possible, or in other words, the "intended method"?</p>
<p>I'll also note that I saw mention of <code>torch.cuda.get/set_rng_state</code>, and maybe I could just do everything I do with the "CPU RNG state" with the "Cuda RNG state" as well; I just wonder if that's the intended use, and didn't see a very direct mention of this in the docs.</p>
 

Latest posts

Top