M-JPEG is Motion Joint Photographic Experts Group
MPEG-4 is Motion Picture Experts Group Version 4
H.264 is Workgroup #H264 of the International Telecom Union

They are compression algorithms for video streams.  M-JPEG is the oldest and least powerful of the 3 whereas H.264 is the newest and most powerful.  MPEG-4 is the most popular as of 2008. 

Uses of MPEG-4 include compression of Audio Video data for web (streaming media) and CD distribution, voice (telephone, videophone) and broadcast television applications. MPEG-4 and H.264 support externally specified Digital Rights Management and various types of interactivity.   The confusing bit is that H.264 is also referred to as MPEG-4 Part 10 or Advanced Video Coding.  H.264 is used in High Definition DVD and Blu-ray due to its higher compression ratio.

M-JPEG compresses each video frame separately whereas MPEG-4 and H.264 ignore the constituents of the image that are the same as the previous frame and compress changes only.  As a result, MPEG-4 can achieve up to 8 times higher compression than M-JPEG.  H.264 improves over MPEG-4 by about one time.