Extract a video frame by time base: CFR, VFR and exact index checks
A runnable Python and FFmpeg frame-extraction recipe with original CFR/VFR fixtures, rational PTS tables, three selection rules, PNG pixel hashes and 16 executed checks.
Select an actual decoded frame using an explicit at, after or zero-based frame-index policy, then verify its timestamp and saved PNG pixels. Arbitrary requested seconds do not necessarily name an existing frame.
Previously recorded numerical observations below describe the original local fixture runs. For a reproduction, use the new receipt and its actual tool versions and hashes; an old observation is not a guaranteed result on every installation.
Preparation
- Save each complete code block below under its displayed filename in one empty local folder. Run commands from that folder. The scripts create their synthetic inputs and manifests in the chosen output directory; inspect those files after running. Use an output directory that does not already exist. All relative paths below refer to this working folder.
- Save both complete Python files below together. The fixture runner generates the original CFR/VFR videos locally.
- Prepare Python 3 and an installed FFmpeg binary. The recorded run used Python 3.12.3 and FFmpeg 6.0 on macOS arm64; no ffprobe or additional Python package is required.
- Set FFMPEG_BIN if ffmpeg is not on PATH. Choose a fresh output directory; the fixture runner and extractor refuse overwrite.
- For your own clip, decide whether time is relative to its first decoded PTS and whether the output should follow display rotation or retain raw orientation. This script uses first-PTS-relative seconds.
01. Build the decoded timestamp inventory
Run the fixture suite before touching a production clip. CFR has ten frames at 0.00, 0.10 … 0.90 s; VFR has eight at 0.00, 0.04, 0.12, 0.28, 0.32, 0.60, 0.64 and 0.92 s. Both measured time bases are 1/1000. The CSV and inventory JSON tie every zero-based decoded index to its integer PTS; seconds equal PTS multiplied by the time base. Do not infer VFR positions from average FPS.
export FFMPEG_BIN=/YOUR/INSTALLED/ffmpeg
python3 run-fixtures.py --out-dir ./my-run
02. Choose at, after or an exact decoded frame index
at selects the latest frame start at or before the requested relative time; after selects the first at or after it. For VFR target 0.20 s, at returns index 2 at 0.12 s, while after returns index 3 at 0.28 s. --rule frame --index 3 also returns 0.28 s; an exact index is not an exact arbitrary timestamp. The script decodes from the beginning and rejects a time outside the first-to-last PTS range rather than inventing the final frame duration.
python3 extract-frame.py --input my-run/vfr.mkv --rule at --time 0.20 --out ./at.png
python3 extract-frame.py --input my-run/vfr.mkv --rule after --time 0.20 --out ./after.png
python3 extract-frame.py --input my-run/vfr.mkv --rule frame --index 3 --out ./index3.png
03. Verify the original PTS and the saved pixels separately
Read the PNG selection receipt and inventory sidecar, not the filename. The source framehash is generated with preserved timestamps, passthrough frame scheduling and an explicit time base; its #tb header defines the integer PTS units. A second decode verifies that the PNG RGB hash matches the selected source frame. The framehash packet-duration column is not used to infer VFR display holds. File SHA-256 identifies bytes; decoded RGB SHA-256 identifies pixels under this conversion and orientation.
python3 -m json.tool at.selection.json
python3 -m json.tool at.inventory.json
04. Choose rotation handling and keep failed checks visible
Display extraction follows default autorotation; raw extraction applies -noautorotate before input. The executed 90-degree metadata fixture produced 180×320 display pixels and 320×180 raw pixels. A nonzero 2-second source start was also tested using relative times. These checks do not cover every display matrix, mirrored source or damaged timestamp stream. Keep the selected policy in the receipt and stop on an ambiguous inventory instead of forcing CFR or interpolation.
python3 extract-frame.py --input my-run/rotation-90.mp4 --rule frame --index 0 --orientation display --out ./display.png
python3 extract-frame.py --input my-run/rotation-90.mp4 --rule frame --index 0 --orientation raw --out ./raw.png
Acceptance
- The suite saved from the code below passes its 16 named scenarios and records the running Python and FFmpeg versions plus the exact SHA-256 hashes of both executed scripts.
- The CFR/VFR table records decoded index, integer PTS and rational time base, including every VFR gap.
- At 0.20 s the VFR at/after outputs have the distinct verified indices and PTS stated in the table; no synthetic 0.20-s frame is substituted.
- The saved PNG decodes to the same RGB hash as the selected source frame under the chosen orientation.
- Negative time, time beyond the last PTS and an out-of-range index stop with an error.
- Rotation dimensions are checked against the explicit display/raw choice; nonzero stream start uses first-PTS-relative time.
- The receipt makes no claim that model quality, all codecs, all display transforms or every real input timestamp has been validated.
Troubleshooting
The frame looks earlier or later than the requested seconds.
Check the selected rule and the actual PTS table. A request between two VFR timestamps legitimately selects different frames for at and after; do not change the reported timestamp to match the request.
A saved PNG is sideways or its dimensions are exchanged.
Inspect the desired display orientation. Compare --orientation display with raw; PNG pixels follow the selected policy, while video rotation side data is not assumed to remain a useful PNG viewing instruction.
The extractor rejects the last part of the video.
The bounded policy ends at the last known PTS and does not invent its display duration. Select an explicit last-frame index or establish that duration separately before extending the policy.
Frame indices disagree with a timestamp derived from FPS.
Use the decoded presentation inventory and the rational time base. Average FPS, packets and keyframes are different identities, especially for VFR.
The hash differs on another FFmpeg installation.
Compare decoded RGB hashes under the same pixel format and orientation before comparing encoded PNG bytes. Record the new tool version; each receipt proves only the run whose versions and hashes it records.
Keep with the result
Keep the PNG, its selection receipt and the source frame table together. The selected rule, actual timestamp and pixel check make the extraction reviewable; they do not manufacture a frame at a timestamp the source never contained.
Complete source and input construction
Save every file below with its exact filename in the same folder. The source includes the input definitions and construction logic; code blocks are complete, including all companion modules.
extract-frame.py
#!/usr/bin/env python3
"""Select one decoded frame by an explicit time/index policy. Local files only."""
import argparse, hashlib, json, os, re, shutil, struct, subprocess, sys
from fractions import Fraction
from pathlib import Path
def filehash(path):
value=hashlib.sha256()
with path.open("rb") as source:
for part in iter(lambda:source.read(1048576), b""):value.update(part)
return value.hexdigest()
def invoke(ffmpeg, args):
return subprocess.run([ffmpeg, '-hide_banner', '-nostdin', '-threads', '1', *args], check=True, capture_output=True, text=True, timeout=90)
def executable():
value=os.environ.get('FFMPEG_BIN') or shutil.which('ffmpeg')
if not value: raise ValueError('Set FFMPEG_BIN to an installed FFmpeg executable (tested: 6.0).')
return value
def inventory(ffmpeg, source):
if not source.is_file() or source.is_symlink(): raise ValueError('Input must be a regular local file, not a URL or symlink.')
run=invoke(ffmpeg,['-copyts','-noautorotate','-i',str(source),'-map','0:v:0','-an','-sn','-dn','-vf','showinfo','-fps_mode','passthrough','-f','null','-'])
match=re.search(r'config in time_base:\s*(\d+)/(\d+)',run.stderr)
if not match: raise ValueError('No decoder/filter time base found; do not infer it from nominal FPS.')
tb=Fraction(int(match[1]),int(match[2])); frames=[]
for item in re.finditer(r'\bn:\s*(\d+)\s+pts:\s*(-?\d+)\s+pts_time:',run.stderr):
n,pts=int(item[1]),int(item[2]); frames.append({'index':n,'pts':pts,'timestamp':Fraction(pts)*tb})
if not frames or len(frames)>100000: raise ValueError('No frames or unsupported inventory size (>100000 frames).')
if [f['index'] for f in frames]!=list(range(len(frames))): raise ValueError('Non-contiguous decoded indices.')
if any(b['timestamp']<=a['timestamp'] for a,b in zip(frames,frames[1:])): raise ValueError('Missing, repeated, or non-monotonic PTS require separate repair/interpretation.')
first=frames[0]['timestamp']
for f in frames: f['relative']=f['timestamp']-first
return tb,frames,run.stderr
def choose(frames,rule,time=None,index=None):
if rule=='frame':
if index is None or index<0 or index>=len(frames): raise ValueError('Zero-based frame index is outside decoded inventory.')
return frames[index]
if time is None: raise ValueError('Time policy needs --time in seconds; accepts a decimal or rational such as 1/5.')
target=Fraction(time)
if target<0 or target>frames[-1]['relative']: raise ValueError('Time outside first-to-last PTS range; final display duration is not inferred.')
candidates=[f for f in frames if f['relative']<=target] if rule=='at' else [f for f in frames if f['relative']>=target]
return candidates[-1] if rule=='at' else candidates[0]
def row(f,tb):
return {'frameIndex':f['index'],'pts':f['pts'],'timeBase':str(tb),'timestampSecondsRational':str(f['timestamp']), 'timestampSeconds':float(f['timestamp']),'relativeSecondsRational':str(f['relative']),'relativeSeconds':float(f['relative'])}
def extract(ffmpeg,source,out,rule,time=None,index=None,orientation='display'):
if out.suffix.lower()!='.png': raise ValueError('Output must use .png.')
out.parent.mkdir(parents=True,exist_ok=True)
receipt=out.with_suffix('.selection.json'); framehash=out.with_suffix('.framehash.txt'); probe=out.with_suffix('.inventory.json')
if any(p.exists() or p.is_symlink() for p in [out,receipt,framehash,probe]): raise ValueError('Refusing to overwrite output or sidecars; select a fresh output name.')
tb,frames,_=inventory(ffmpeg,source); selected=choose(frames,rule,time,index)
filter_=f"select=eq(n\\,{selected['index']}),format=rgb24"
orient=[] if orientation=='display' else ['-noautorotate']
common=['-copyts',*orient,'-i',str(source),'-map','0:v:0','-an','-sn','-dn','-vf',filter_,'-fps_mode','passthrough','-frames:v','1']
invoke(ffmpeg,[*common,'-c:v','png','-threads','1','-f','image2','-update','1','-n',str(out)])
hashed=invoke(ffmpeg,[*common,'-c:v','rawvideo','-enc_time_base',f'{tb.numerator}:{tb.denominator}','-f','framehash','-hash','sha256','-'])
data=[line for line in hashed.stdout.splitlines() if line and not line.startswith('#')]
if len(data)!=1: raise ValueError('Selected framehash did not produce exactly one frame.')
pixelhash=data[0].split(',')[-1].strip()
pnghash=invoke(ffmpeg,['-i',str(out),'-map','0:v:0','-vf','format=rgb24','-frames:v','1','-c:v','rawvideo','-f','framehash','-hash','sha256','-'])
pngdata=[line for line in pnghash.stdout.splitlines() if line and not line.startswith('#')]
if len(pngdata)!=1 or pngdata[0].split(',')[-1].strip()!=pixelhash: raise ValueError('PNG decoded RGB hash does not match selected source frame.')
result={'schemaVersion':1,'policy':rule,'requestedTime':time,'requestedIndex':index,'orientationPolicy':orientation,'selection':row(selected,tb),'inputSha256':filehash(source),'outputPngSha256':filehash(out),'decodedRgbSha256':pixelhash,'pixelMatchVerified':True,'outputDimensions':list(struct.unpack('>II',out.read_bytes()[16:24])),'timeOrigin':'relative to first decoded presentation timestamp','lastFrameDurationInferred':False}
framehash.write_text(hashed.stdout);probe.write_text(json.dumps([row(f,tb) for f in frames],indent=2)+'\n');receipt.write_text(json.dumps(result,indent=2)+'\n')
return result
def main():
p=argparse.ArgumentParser(description=__doc__)
p.add_argument('--input',required=True,type=Path);p.add_argument('--out',required=True,type=Path)
p.add_argument('--rule',required=True,choices=['at','after','frame']);p.add_argument('--time');p.add_argument('--index',type=int)
p.add_argument('--orientation',choices=['display','raw'],default='display');a=p.parse_args()
if a.rule=='frame' and a.time is not None: p.error('frame uses --index, not --time')
if a.rule!='frame' and a.index is not None: p.error('time policies use --time, not --index')
try: print(json.dumps(extract(executable(),a.input,a.out,a.rule,a.time,a.index,a.orientation),indent=2))
except (ValueError,subprocess.CalledProcessError,subprocess.TimeoutExpired) as e: print(f'Extraction stopped: {e}',file=sys.stderr);return 2
return 0
if __name__=='__main__': sys.exit(main())
run-fixtures.py
#!/usr/bin/env python3
"""Create original CFR/VFR fixtures and execute timestamp-selection checks."""
import argparse, hashlib, importlib.util, json, os, platform, shutil, struct, subprocess, sys
from datetime import datetime,timezone
from pathlib import Path
from fractions import Fraction
sys.dont_write_bytecode=True
here=Path(__file__).resolve().parent
spec=importlib.util.spec_from_file_location('selector',here/'extract-frame.py');selector=importlib.util.module_from_spec(spec);spec.loader.exec_module(selector)
def run(work):
ffmpeg=selector.executable()
if work.exists(): raise ValueError('Output directory already exists; use a fresh --out-dir to preserve prior results.')
work.mkdir(parents=True); commands=[];scenarios=[]
script_hashes={name:hashlib.sha256((here/name).read_bytes()).hexdigest() for name in ['extract-frame.py','run-fixtures.py']}
def media(args):
commands.append(['ffmpeg',*[str(x).replace(str(work),'OUTPUT') for x in args]]);return selector.invoke(ffmpeg,args)
def raw(n):
# Unique coloured frame-index tiles and moving stripe; original deterministic RGB data, no third-party picture.
data=bytearray()
for i in range(n):
for y in range(180):
for x in range(320):
color=(20+i*15,38+(x//32)*12,65+(y//30)*18)
if 20+i*20<=x<34+i*20 and 25<=y<155:color=(235,173,86)
if 228<=x<306 and 18<=y<68:
bit=(x-228)//13
color=(208,235,225) if ((i>>bit)&1) else (34,62,78)
data.extend(color)
return bytes(data)
for name,pts,rate in [('cfr',list(range(0,1000,100)),10),('vfr',[0,40,120,280,320,600,640,920],25)]:
source=work/(name+'.rgb');source.write_bytes(raw(len(pts)))
args=['-f','rawvideo','-pixel_format','rgb24','-video_size','320x180','-framerate',str(rate),'-i',str(source),'-an']
if name=='vfr':
expr=str(pts[-1])
for n in range(len(pts)-2,-1,-1):expr=f'if(eq(N,{n}),{pts[n]},{expr})'
args+=['-vf','settb=1/1000,setpts='+expr.replace(',','\\,')]
args+=['-c:v','ffv1','-level','3','-threads','1','-enc_time_base','1:1000','-fps_mode','passthrough','-n',str(work/(name+'.mkv'))]
media(args);source.unlink()
tb,frames,log=selector.inventory(ffmpeg,work/(name+'.mkv'));assert tb==Fraction(1,1000)
assert [f['pts'] for f in frames]==pts
(work/(name+'-inventory.json')).write_text(json.dumps([selector.row(f,tb) for f in frames],indent=2)+'\n')
(work/(name+'-showinfo.txt')).write_text(log.replace(str(work),'OUTPUT'))
hashed=media(['-copyts','-noautorotate','-i',str(work/(name+'.mkv')),'-map','0:v:0','-an','-vf','format=rgb24','-c:v','rawvideo','-enc_time_base','1:1000','-fps_mode','passthrough','-f','framehash','-hash','sha256','-'])
(work/(name+'-all.framehash.txt')).write_text(hashed.stdout);scenarios.append(name+'-pts-inventory')
cases=[('cfr','at','0.25',None,2,200),('cfr','after','0.25',None,3,300),('cfr','frame',None,3,3,300),('vfr','at','0.20',None,2,120),('vfr','after','0.20',None,3,280),('vfr','frame',None,3,3,280),('vfr','at','0.28',None,3,280),('vfr','after','0.28',None,3,280)]
records=[]
for i,(name,rule,time,index,n,pts) in enumerate(cases):
out=work/f'{name}-{rule}-{i}.png';r=selector.extract(ffmpeg,work/(name+'.mkv'),out,rule,time,index)
assert r['selection']['frameIndex']==n and r['selection']['pts']==pts and r['pixelMatchVerified'];records.append(r);scenarios.append(f'{name}-{rule}-{time if time is not None else index}')
_,f,_=selector.inventory(ffmpeg,work/'vfr.mkv')
for label,rule,time,index in [('negative-time','at','-0.1',None),('beyond-last-pts','after','1.1',None),('frame-out-of-range','frame',None,8)]:
try:selector.choose(f,rule,time,index)
except ValueError:scenarios.append(label)
else:raise AssertionError(label)
# A real rotation metadata case verifies extraction policy without implying every matrix is a simple 90-degree rotation.
media(['-i',str(work/'cfr.mkv'),'-map','0:v:0','-an','-c:v','libx264','-crf','0','-pix_fmt','yuv444p','-threads','1','-n',str(work/'rotation-base.mp4')])
media(['-i',str(work/'rotation-base.mp4'),'-map','0:v:0','-c','copy','-metadata:s:v:0','rotate=90','-n',str(work/'rotation-90.mp4')]);(work/'rotation-base.mp4').unlink()
for orientation,dims in [('display',(180,320)),('raw',(320,180))]:
out=work/f'rotation-{orientation}.png';r=selector.extract(ffmpeg,work/'rotation-90.mp4',out,'frame',index=0,orientation=orientation)
assert struct.unpack('>II',out.read_bytes()[16:24])==dims
records.append(r);scenarios.append('rotation-'+orientation)
# Starting with a stream offset verifies that CLI times are relative to the first decoded PTS, not container absolute times.
media(['-copyts','-itsoffset','2','-i',str(work/'vfr.mkv'),'-map','0:v:0','-c','copy','-n',str(work/'vfr-offset.mkv')])
r=selector.extract(ffmpeg,work/'vfr-offset.mkv',work/'vfr-offset-after.png','after',time='0.20');assert r['selection']['pts']==2280 and r['selection']['relativeSecondsRational']=='7/25';records.append(r);scenarios.append('nonzero-start-relative-time')
tool=media(['-version']).stdout
(work/'tool-version.txt').write_text(tool)
tables={}
for name in ['cfr','vfr']:
tables[name]=json.loads((work/(name+'-inventory.json')).read_text())
with (work/'timestamp-table.csv').open('w') as out:
out.write('fixture,frame_index,pts,time_base,timestamp_seconds,relative_seconds\n')
for name,rows in tables.items():
for f in rows:out.write(f"{name},{f['frameIndex']},{f['pts']},{f['timeBase']},{f['timestampSeconds']},{f['relativeSeconds']}\n")
assert script_hashes=={name:hashlib.sha256((here/name).read_bytes()).hexdigest() for name in script_hashes}, 'Scripts changed during fixture execution'
receipt={'schemaVersion':1,'scriptHashes':script_hashes,'recipe':'frame-extraction-timebase-validation','executedAt':datetime.now(timezone.utc).isoformat(),'pythonVersion':platform.python_version(),'platform':platform.system()+' '+platform.machine(),'ffmpegVersion':tool.splitlines()[0],'ffmpegBinarySha256':hashlib.sha256(Path(ffmpeg).read_bytes()).hexdigest(),'sourceType':'original synthetic RGB tiles; FFV1 CFR/VFR and H.264 rotation fixture','noModelGeneration':True,'productionTaskRun':False,'fixtureOnly':True,'passedScenarios':scenarios,'selectionRecords':records,'timestampTables':tables,'commands':commands,'checksums':{str(p.relative_to(work)):hashlib.sha256(p.read_bytes()).hexdigest() for p in sorted(work.iterdir()) if p.is_file()}}
(work/'execution-receipt.json').write_text(json.dumps(receipt,indent=2)+'\n');print(json.dumps({'passed':len(scenarios),'scenarios':scenarios,'ffmpegVersion':receipt['ffmpegVersion'],'fixtureBytes':sum(p.stat().st_size for p in work.iterdir())},indent=2))
if __name__=='__main__':
parser=argparse.ArgumentParser(description=__doc__);parser.add_argument('--out-dir',type=Path,default=here/'run-output');args=parser.parse_args()
try:run(args.out_dir.resolve())
except Exception as e:print(f'Fixture failed: {e}',file=sys.stderr);sys.exit(1)
Technical references
