OiO.lk Community platform!

Oio.lk is an excellent forum for developers, providing a wide range of resources, discussions, and support for those in the developer community. Join oio.lk today to connect with like-minded professionals, share insights, and stay updated on the latest trends and technologies in the development field.
  You need to log in or register to access the solved answers to this problem.
  • You have reached the maximum number of guest views allowed
  • Please register below to remove this limitation

Init Script on databricks

  • Thread starter Thread starter Greencolor
  • Start date Start date
G

Greencolor

Guest
Life of the Init script from DBFS has come to an end. I used to deploy the databricks cluster using this powershell script with InitScript paramter.

Code:
New-DatabricksCluster -BearerToken $ADB_Token -Region $region -ClusterName $cname -SparkVersion $csparkV `
-NodeType $cnodeT -MinNumberOfWorkers $cminWorker -MaxNumberOfWorkers $cmaxWorker -AutoTerminationMinutes $cterm `
-InitScripts "/Init/pyodbc.sh" -UniqueNames -Update

But oficial documentation of this powershell function states that it must but DBFS path.

.PARAMETER InitScripts Init scripts to run post creation. As array of strings - paths must be full dbfs paths. Example: "dbfs:/script/script1", "dbfs:/script/script2"

Now I want to migrate from DBFS to Workspace file location, while I can achieve this via databricks UI, i was wondering how can I do the same thing with the powershell code or even databricks API.
<p>Life of the Init script from DBFS has come to an end. I used to deploy the databricks cluster using this powershell script with <code>InitScript</code> paramter.</p>
<pre><code>New-DatabricksCluster -BearerToken $ADB_Token -Region $region -ClusterName $cname -SparkVersion $csparkV `
-NodeType $cnodeT -MinNumberOfWorkers $cminWorker -MaxNumberOfWorkers $cmaxWorker -AutoTerminationMinutes $cterm `
-InitScripts "/Init/pyodbc.sh" -UniqueNames -Update
</code></pre>
<p><a href="https://www.powershellgallery.com/p....1.5/Content/Public\New-DatabricksCluster.ps1" rel="nofollow noreferrer">But oficial documentation of this powershell</a> function states that it must but DBFS path.</p>
<blockquote>
<p>.PARAMETER InitScripts Init scripts to run post creation. As array of
strings - <strong>paths must be full dbfs paths</strong>. Example:
"dbfs:/script/script1", "dbfs:/script/script2"</p>
</blockquote>
<p>Now I want to migrate from DBFS to Workspace file location, while I can achieve this via databricks UI, i was wondering how can I do the same thing with the powershell code or even databricks API.</p>
Continue reading...
 

Latest posts

A
Replies
0
Views
1
Ahmed Mansour
A
D
Replies
0
Views
1
Domantas Giedraitis
D
S
Replies
0
Views
1
Srikanta saw
S
Top