When WithLatestFrom operator is not enough

Question

I just stumbled upon a situation where I'm not being able to avoid keeping state, but I still get the feeling that this could be achieved using only Rx:

I've got two observables, the main one invokes the onNext every time a user inserts a value:

-- RV -- RV -- RV -- IV ------ ... ->

The other one invokes onNext every the user wants to remove the last value

------------------------- X -- ... ->

Now the thing is, I'm only concerned when the user removes a Relevant value, so in this case I wanted the following sequence:

-- RV -- RV -- RV ------------ ... ->

At this point I'm able to get what I want simply by using the WithLatestFrom operator, however in the following case:

-- RV -- RV -- RV -- IV --------- RV ----- ... ->

------------------------- X -- X ---- X -- ... ->

I don't know how I can create a observable who returns me the following stream:

-- RV -- RV -- RV ------------ X ---- X -- ... ->

Caption: RV = Relevant value IV = Irrelevant value