AV_SEEK_FRAME setting at start failing for compressed formats like .wmv .mp4(h264)

Question

I have been trying to decode videos where I need to offset start position. I have tried to use AV_seek_frame but it works on raw video format (.y4m) but fails on mpeg2 compressed formats. Generally videos are in mp4 or wmv and I need to write for those directly instead of converting them into .y4m

Tried with avformat_seek_file api too but still failing?

Need help.. Here how I’m using it

//When using with time 
   double t = 5;//time in seconds
   int64_t timestamp = t * AV_TIME_BASE; //destination time
   AVRational r = av_make_q(1, AV_TIME_BASE);      
   if (VideoStreamIndex >= 0) {
          timestamp = av_rescale_q(timestamp, r,
                 fmt_ctx->streams[VideoStreamIndex]->time_base);
   }

   av_seek_frame(fmt_ctx, VideoStreamIndex, timestamp, AVSEEK_FLAG_ANY);

or if I know the frame no

int frameno = 100;
int64_t point = (int64_t(frameno) * pavStream->r_frame_rate.den *  pavStream->time_base.den) / (int64_t(pavStream->r_frame_rate.num) *pavStream->time_base.num);

av_seek_frame(fmt_ctx, VideoStreamIndex, point, 0);