How to interpolate missing values with groupby?

Question

I cannot get missing values to interpolate correctly when I use the groupby function.

Here is a quick example of what I have tried:

import pandas as pd
import numpy as np

# Create data
state = pd.Series(['A', 'A', 'A', 'A', 'B', 'B', 'B', 'B'])
population = pd.Series([100, 150, np.nan, np.nan, 50, 125, np.nan, np.nan])
year = [2016, 2017, 2018, 2019, 2016, 2017, 2018, 2019]
dict = {'state': state, 'population': population, 'year': year}  
df = pd.DataFrame(dict) 

# Interpolate population, grouped by states
df.groupby('state').apply(lambda x: x.interpolate(method='linear')) 

  state  population  year
0     A       100.0  2016
1     A       150.0  2017
2     A       150.0  2018
3     A       150.0  2019
4     B        50.0  2016
5     B       125.0  2017
6     B       125.0  2018
7     B       125.0  2019

As you notice, when grouping by state, it is simply repeating the last value.

Answer 1

And base on what you need , pass the method spline

df.groupby('state')['population'].apply(lambda x : x.interpolate(method = "spline", order = 1, limit_direction = "both"))
0    100.0
1    150.0
2    200.0
3    250.0
4     50.0
5    125.0
6    200.0
7    275.0
Name: population, dtype: float64