OiO.lk Community platform!

Oio.lk is an excellent forum for developers, providing a wide range of resources, discussions, and support for those in the developer community. Join oio.lk today to connect with like-minded professionals, share insights, and stay updated on the latest trends and technologies in the development field.
  You need to log in or register to access the solved answers to this problem.
  • You have reached the maximum number of guest views allowed
  • Please register below to remove this limitation

Efficiently Resampling and Interpolating Pandas DataFrames with Millisecond Accuracy

  • Thread starter Thread starter bfgt
  • Start date Start date
B

bfgt

Guest
I have a Pandas DataFrame with timestamps that have millisecond accuracy and corresponding altitude values. I want to resample and interpolate this data efficiently. Here is a simple example:

Code:
import pandas as pd
import numpy as np

# Generate 5 random timestamps within the same minute with millisecond accuracy
base_timestamp = pd.Timestamp.now().floor("min")  # Get the current time, floored to the nearest minute
timestamps = [
    base_timestamp + pd.to_timedelta(np.random.randint(0, 60000), unit="ms")
    for _ in range(5)
]

# Generate random altitudes
altitudes = np.random.uniform(100, 1000, size=5)  # Random altitudes between 100 and 1000

# Create the DataFrame
df = pd.DataFrame({"timestamp": timestamps, "altitude": altitudes}).sort_values("timestamp")

A method that works but is terribly inefficient is the following:

Code:
df_interpolated = (
    df.set_index("timestamp").resample("1ms").interpolate().resample("1s").interpolate()
)

Plotting the results shows that it works:

Code:
import plotly.graph_objects as go

fig = go.Figure()
fig.add_trace(go.Scatter(x=df.timestamp, y=df.altitude, mode="lines", name="Original"))

fig.add_trace(
    go.Scatter(
        x=df_interpolated.index,
        y=df_interpolated.altitude,
        mode="markers",
        name="Interpolated 1ms and 1s",
    )
)
fig.show()

results

Any idea on how to do this better?
<p>I have a Pandas DataFrame with timestamps that have millisecond accuracy and corresponding altitude values. I want to resample and interpolate this data efficiently. Here is a simple example:</p>
<pre><code>import pandas as pd
import numpy as np

# Generate 5 random timestamps within the same minute with millisecond accuracy
base_timestamp = pd.Timestamp.now().floor("min") # Get the current time, floored to the nearest minute
timestamps = [
base_timestamp + pd.to_timedelta(np.random.randint(0, 60000), unit="ms")
for _ in range(5)
]

# Generate random altitudes
altitudes = np.random.uniform(100, 1000, size=5) # Random altitudes between 100 and 1000

# Create the DataFrame
df = pd.DataFrame({"timestamp": timestamps, "altitude": altitudes}).sort_values("timestamp")

</code></pre>
<p>A method that works but is terribly inefficient is the following:</p>
<pre><code>df_interpolated = (
df.set_index("timestamp").resample("1ms").interpolate().resample("1s").interpolate()
)
</code></pre>
<p>Plotting the results shows that it works:</p>
<pre><code>import plotly.graph_objects as go

fig = go.Figure()
fig.add_trace(go.Scatter(x=df.timestamp, y=df.altitude, mode="lines", name="Original"))

fig.add_trace(
go.Scatter(
x=df_interpolated.index,
y=df_interpolated.altitude,
mode="markers",
name="Interpolated 1ms and 1s",
)
)
fig.show()
</code></pre>
<p><a href="https://i.sstatic.net/tbDuYuyf.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/tbDuYuyf.png" alt="results" /></a></p>
<p>Any idea on how to do this better?</p>
 
Top